Sep 10 22:56:14.761682 ip-10-0-138-150 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 22:56:14.761695 ip-10-0-138-150 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 22:56:14.761705 ip-10-0-138-150 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 22:56:14.761898 ip-10-0-138-150 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 22:56:25.010613 ip-10-0-138-150 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 22:56:25.010652 ip-10-0-138-150 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 99c6f76f00324746bf1c737980fd24e8 --
Sep 10 22:58:03.453326 ip-10-0-138-150 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 22:58:03.888578 ip-10-0-138-150 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 22:58:03.888578 ip-10-0-138-150 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 22:58:03.888578 ip-10-0-138-150 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 22:58:03.888578 ip-10-0-138-150 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 22:58:03.888578 ip-10-0-138-150 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 22:58:03.890458 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.890362    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 22:58:03.894341 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894321    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:58:03.894341 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894341    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894346    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894349    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894352    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894355    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894358    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894361    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894364    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894367    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894369    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894372    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894374    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894381    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894383    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894386    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894388    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894391    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894394    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894396    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:58:03.894415 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894399    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894401    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894404    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894407    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894409    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894412    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894415    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894417    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894420    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894423    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894426    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894428    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894431    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894434    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894437    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894440    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894442    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894445    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894448    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894452    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:58:03.894905 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894454    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894457    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894459    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894462    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894464    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894467    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894469    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894472    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894474    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894479    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894483    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894485    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894488    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894490    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894493    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894497    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894499    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894502    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894504    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894507    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:58:03.895381 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894509    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894512    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894516    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894520    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894523    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894526    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894528    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894531    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894534    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894536    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894540    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894543    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894545    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894548    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894550    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894553    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894555    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894558    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894562    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:58:03.895933 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894564    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894567    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894569    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894572    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894575    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894578    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894581    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894584    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894587    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894589    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894592    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894594    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894597    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894600    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894602    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894605    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894609    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894612    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894614    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:58:03.896390 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.894617    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:58:03.897191 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897176    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:58:03.897191 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897190    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897193    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897196    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897199    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897203    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897206    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897208    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897211    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897216    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897220    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897223    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897226    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897229    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897231    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897234    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897236    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897239    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897241    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897244    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897247    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:58:03.897251 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897249    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897252    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897255    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897258    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897261    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897264    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897266    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897269    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897272    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897274    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897277    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897280    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897282    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897285    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897288    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897290    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897293    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897296    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897298    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897301    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:58:03.897764 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897303    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897306    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897309    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897311    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897314    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897317    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897320    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897326    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897329    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897333    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897337    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897340    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897343    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897346    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897350    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897353    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897356    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897359    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897361    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:58:03.898271 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897364    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897367    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897369    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897372    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897374    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897377    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897379    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897382    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897384    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897387    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897390    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897393    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897395    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897398    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897400    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897403    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897406    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897408    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897411    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:58:03.898759 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897413    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897418    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897421    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897423    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897426    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897428    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897431    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897434    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897436    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897439    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897443    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897446    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897449    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897452    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897462    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897465    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897468    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897470    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897473    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897476    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:58:03.899261 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897911    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897917    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897920    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897923    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897926    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897930    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897932    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897935    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897938    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897941    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897943    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897946    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897949    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897953    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897956    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897959    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897962    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897964    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897967    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897970    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:58:03.899868 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897973    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897975    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897978    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897981    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897983    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897985    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897988    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897990    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897993    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897995    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.897998    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898000    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898003    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898005    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898008    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898011    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898014    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898016    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898019    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898022    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:58:03.900373 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898024    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898027    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898029    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898032    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898034    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898038    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898041    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898044    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898046    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898049    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898051    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898054    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898057    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898059    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898063    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898067    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898070    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898073    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898075    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:58:03.900889 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898078    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898081    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898083    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898086    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898088    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898091    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898095    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898098    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898101    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898104    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898107    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898110    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898112    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898115    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898117    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898119    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898122    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898125    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898129    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898131    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:58:03.901404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898134    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898136    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898139    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898141    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898144    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898146    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898149    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898152    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898154    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898157    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898159    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898161    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898164    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898167    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898169    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898176    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898179    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898181    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898186    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898189    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:58:03.901931 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898272    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898310    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898318    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898323    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898328    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898331    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898335    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898340    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898343    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898346    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898351    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898354    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898357    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898360    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898363    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898366    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898369    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898372    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898376    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898379    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898382    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898385    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898389    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898393    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 22:58:03.902428 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898396    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898399    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898402    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898405    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898408    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898411    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898414    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898418    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898422    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898427    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898430    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898433    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898436    2577 flags.go:64] FLAG: --enable-server="true"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898440    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898445    2577 flags.go:64] FLAG: --event-burst="100"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898448    2577 flags.go:64] FLAG: --event-qps="50"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898451    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898454    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898458    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898462    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898465    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898468    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898471    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898474    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898477    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 22:58:03.903044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898480    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898483    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898486    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898489    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898492    2577 flags.go:64] FLAG: --feature-gates=""
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898496    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898499    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898503    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898506    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898509    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898512    2577 flags.go:64] FLAG: --help="false"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898515    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-138-150.ec2.internal"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898518    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898521    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898524    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898528    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898533    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898536    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898539    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898542    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898545    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898548    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898552    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898555    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 10 22:58:03.903677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898558    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898560    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898564    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898567    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898570    2577 flags.go:64] FLAG: --lock-file=""
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898573    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898576    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898578    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898584    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898587    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898590    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898593    2577 flags.go:64] FLAG: --logging-format="text"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898596    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898599    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898602    2577 flags.go:64] FLAG: --manifest-url=""
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898605    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898609    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898612    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898617    2577 flags.go:64] FLAG: --max-pods="110"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898620    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898622    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898625    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898628    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898631    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898636    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 22:58:03.904325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898639    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898646    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898650    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898653    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898657    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898660    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898666    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898669    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898672    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898683    2577 flags.go:64] FLAG: --port="10250"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898688    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898691    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c07c09fb096e4a92"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898694    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898697    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898700    2577 flags.go:64] FLAG: --register-node="true"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898703    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898707    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898710    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898713    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898716    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898720    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898723    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898726    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898728    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898732    2577 flags.go:64] FLAG: --runonce="false"
Sep 10 22:58:03.905194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898734    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898737    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898740    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898743    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898746    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898751    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898755    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898758    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898761    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898764    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898767    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898770    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898773    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898791    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898795    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898800    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898803    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898806    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898810    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898813    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898816    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898819    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898822    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898825    2577 flags.go:64] FLAG: --v="2"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898829    2577 flags.go:64] FLAG: --version="false"
Sep 10 22:58:03.905947 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898833    2577 flags.go:64] FLAG: --vmodule=""
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898837    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.898840    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898959    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898962    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898965    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898968    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898971    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898973    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898976    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898978    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898981    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898986    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898990    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898993    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898995    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.898998    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899000    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899003    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899006    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:58:03.906592 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899009    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899012    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899015    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899018    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899020    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899023    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899025    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899028    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899031    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899033    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899036    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899039    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899041    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899044    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899047    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899049    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899052    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899055    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899057    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899060    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:58:03.907478 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899063    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899065    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899068    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899070    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899074    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899079    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899081    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899084    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899087    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899090    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899092    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899095    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899097    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899101    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899104    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899107    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899110    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899114    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899117    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899120    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:58:03.908348 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899123    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899125    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899128    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899131    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899134    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899136    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899139    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899142    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899144    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899147    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899150    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899152    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899155    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899158    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899160    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899163    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899169    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899173    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899177    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:58:03.909205 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899180    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899183    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899187    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899189    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899192    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899195    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899204    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899207    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899210    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899213    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899216    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899219    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899221    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899224    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899227    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899229    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899232    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899235    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899237    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899240    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:58:03.909707 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899243    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899245    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.899248    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.900006    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.909195    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.909220    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.910089    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.910103    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910207    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910216    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910224    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910230    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910235    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910241    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:58:03.910393 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910245    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910249    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910254    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910258    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910262    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910266    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910271    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910276    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910280    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910284    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910288    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910293    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910297    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910301    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910305    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910310    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910314    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910318    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910323    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910327    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:58:03.911118 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910332    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910337    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910342    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910348    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910352    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910357    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910361    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910365    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910369    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910374    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910378    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910382    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910387    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910391    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910395    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910398    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910402    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910407    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910411    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910416    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:58:03.911897 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910420    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910424    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910429    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910433    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910437    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910442    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910446    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910450    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910456    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910460    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910464    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910468    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910473    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910477    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910483    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910490    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910495    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910500    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910505    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:58:03.912453 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910509    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910514    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910519    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910524    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910528    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910532    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910537    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910541    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910545    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910549    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910554    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910557    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910564    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910571    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910577    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910582    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910587    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910591    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910595    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:58:03.913248 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910599    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910603    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910607    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910611    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910616    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910620    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910624    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910628    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910633    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910637    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910641    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910645    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910650    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910655    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910659    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:58:03.914011 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.910667    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910908    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910919    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910924    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910930    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910934    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910939    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910943    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910947    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910951    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910956    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910961    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910966    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910972    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910976    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910980    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910985    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910989    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910993    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.910997    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 22:58:03.914385 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911001    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911006    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911010    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911014    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911018    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911022    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911026    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911031    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911035    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911039    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911044    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911051    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911058    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911063    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911067    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911072    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911076    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911080    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911084    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911088    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 22:58:03.915110 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911092    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911097    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911102    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911106    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911111    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911114    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911119    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911123    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911127    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911131    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911135    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911140    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911144    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911148    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911152    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911157    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911161    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911165    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911170    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911175    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 22:58:03.915608 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911180    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911184    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911188    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911192    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911197    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911201    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911205    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911212    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911217    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911222    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911226    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911231    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911235    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911239    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911244    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911248    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911252    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911259    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911264    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 22:58:03.916177 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911270    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911274    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911279    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911283    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911287    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911292    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911296    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911300    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911304    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911309    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911313    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911317    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911321    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911325    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911330    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911334    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911338    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911342    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911347    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911351    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 22:58:03.916655 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:03.911355    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 22:58:03.917258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.911363    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 22:58:03.917258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.911560    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 22:58:03.917258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.914665    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 22:58:03.917258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.915880    2577 server.go:1019] "Starting client certificate rotation"
Sep 10 22:58:03.917258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.915987    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 22:58:03.917258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.916021    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 22:58:03.918738 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.918713    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 22:58:03.919981 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.919958    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 22:58:03.932009 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.931988    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 10 22:58:03.936209 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.936191    2577 log.go:25] "Validated CRI v1 image API"
Sep 10 22:58:03.937181 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.937162    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 22:58:03.940518 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.940497    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 81d1407e-37f0-47ca-877e-0c5e766fc364:/dev/nvme0n1p3 b42deeee-144d-4166-a8d5-57976e3292fd:/dev/nvme0n1p4]
Sep 10 22:58:03.940594 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.940517    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 22:58:03.947640 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.947504    2577 manager.go:216] Machine: {Timestamp:2026-09-10 22:58:03.946064007 +0000 UTC m=+0.382118772 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3138360 MemoryCapacity:33164480512 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27e296a8bb78ac351e5bd082f78db9 SystemUUID:ec27e296-a8bb-78ac-351e-5bd082f78db9 BootID:99c6f76f-0032-4746-bf1c-737980fd24e8 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:aa:40:79:19:91 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:aa:40:79:19:91 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:6e:2e:e6:4b:31:79 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164480512 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 22:58:03.947640 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.947618    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 22:58:03.947825 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.947715    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 22:58:03.949673 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.949634    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 22:58:03.949827 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.949673    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-150.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 22:58:03.950577 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.950564    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 22:58:03.950632 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.950583    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 22:58:03.950632 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.950594    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 22:58:03.950724 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.950710    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 22:58:03.951452 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.951439    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 22:58:03.952669 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.952657    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 22:58:03.956270 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956257    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 22:58:03.956403 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956393    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 22:58:03.956444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956425    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 22:58:03.956444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956440    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 22:58:03.956510 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956459    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 22:58:03.956510 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956475    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 10 22:58:03.956510 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956488    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 22:58:03.956886 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956872    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 22:58:03.956938 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.956895    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 22:58:03.960946 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.960929    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 22:58:03.964229 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.964213    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 22:58:03.964281 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.964245    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 22:58:03.966949 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.966931    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 22:58:03.967025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.966983    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 22:58:03.967025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.966991    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 22:58:03.967025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.966999    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 22:58:03.967025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967008    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 22:58:03.967025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967013    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 22:58:03.967025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967027    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 22:58:03.967213 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967047    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 22:58:03.967213 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967055    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 22:58:03.967213 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967060    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 22:58:03.967213 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967070    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 22:58:03.967213 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.967086    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 22:58:03.968124 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.968112    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 22:58:03.968178 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.968126    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 22:58:03.968178 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:03.968148    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 22:58:03.968178 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:03.968151    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 22:58:03.972303 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.972289    2577 server.go:1268] "Started kubelet"
Sep 10 22:58:03.972431 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.972375    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 22:58:03.972506 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.972456    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 22:58:03.972876 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.972842    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbwtg"
Sep 10 22:58:03.972876 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.972820    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 22:58:03.973303 ip-10-0-138-150 systemd[1]: Started Kubernetes Kubelet.
Sep 10 22:58:03.974331 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.974120    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 22:58:03.976669 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.976650    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 22:58:03.977277 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:03.977258    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 22:58:03.981137 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.981119    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-150.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 22:58:03.982153 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:03.982134    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-hbwtg"
Sep 10 22:58:03.982296 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:03.981118    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-150.ec2.internal.18d417c4548985bf  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-150.ec2.internal,UID:ip-10-0-138-150.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-150.ec2.internal,},FirstTimestamp:2026-09-10 22:58:03.972257215 +0000 UTC m=+0.408311980,LastTimestamp:2026-09-10 22:58:03.972257215 +0000 UTC m=+0.408311980,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-150.ec2.internal,}"
Sep 10 22:58:04.071792 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.071744    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 22:58:04.072424 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.072402    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 22:58:04.072595 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.072555    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.074073 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.073615    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 22:58:04.074073 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.073655    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 22:58:04.074073 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.073674    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 22:58:04.074073 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.073850    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 22:58:04.074073 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.073859    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 22:58:04.074804 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.074610    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 22:58:04.074804 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.074613    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-150.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 22:58:04.074994 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.074979    2577 factory.go:55] Registering systemd factory
Sep 10 22:58:04.075046 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.075000    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 10 22:58:04.075275 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.075255    2577 factory.go:153] Registering CRI-O factory
Sep 10 22:58:04.075275 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.075277    2577 factory.go:223] Registration of the crio container factory successfully
Sep 10 22:58:04.075406 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.075338    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 22:58:04.075406 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.075361    2577 factory.go:103] Registering Raw factory
Sep 10 22:58:04.075406 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.075377    2577 manager.go:1196] Started watching for new ooms in manager
Sep 10 22:58:04.075832 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.075819    2577 manager.go:349] Starting recovery of all containers
Sep 10 22:58:04.080868 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.080810    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 22:58:04.085917 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.085896    2577 manager.go:354] Recovery completed
Sep 10 22:58:04.090071 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.090053    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:58:04.092330 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.092306    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:58:04.092450 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.092344    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:58:04.092450 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.092358    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:58:04.093292 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.093272    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 22:58:04.093292 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.093288    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 22:58:04.093420 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.093307    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 22:58:04.093420 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.093307    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:58:04.094052 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.094035    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:58:04.094152 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.094064    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:58:04.094152 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.094074    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:58:04.094477 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.094464    2577 policy_none.go:49] "None policy: Start"
Sep 10 22:58:04.094477 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.094479    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 22:58:04.094588 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.094491    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 22:58:04.095386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.095371    2577 policy_none.go:47] "Start"
Sep 10 22:58:04.140705 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.140435    2577 manager.go:337] "Starting Device Plugin manager"
Sep 10 22:58:04.140705 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.140686    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 22:58:04.140705 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.140698    2577 server.go:87] "Starting device plugin registration server"
Sep 10 22:58:04.141019 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.140949    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 22:58:04.141019 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.140961    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 22:58:04.141123 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.141094    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 22:58:04.141202 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.141190    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 22:58:04.141202 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.141202    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 22:58:04.141717 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.141686    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 22:58:04.141818 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.141746    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.207836 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.207802    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 22:58:04.207984 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.207862    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 22:58:04.207984 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.207887    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 22:58:04.208082 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.207978    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 22:58:04.210145 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.210120    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:58:04.242042 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.241999    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:58:04.243122 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.243103    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:58:04.243256 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.243132    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:58:04.243256 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.243143    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:58:04.243256 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.243169    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.251558 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.251531    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.251558 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.251562    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-150.ec2.internal\": node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.268111 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.268083    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.308082 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.308051    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal"]
Sep 10 22:58:04.308225 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.308130    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:58:04.309147 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.309128    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:58:04.309251 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.309158    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:58:04.309251 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.309167    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:58:04.310287 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.310275    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:58:04.310409 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.310394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.310451 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.310426    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:58:04.311079 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.311062    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:58:04.311172 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.311092    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:58:04.311172 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.311104    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:58:04.311172 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.311067    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:58:04.311172 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.311162    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:58:04.311172 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.311171    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:58:04.312323 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.312308    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.312379 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.312343    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 22:58:04.313074 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.313058    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 22:58:04.313173 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.313088    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 22:58:04.313173 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.313101    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeHasSufficientPID"
Sep 10 22:58:04.336510 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.336482    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-150.ec2.internal\" not found" node="ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.339841 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.339824    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-150.ec2.internal\" not found" node="ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.369133 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.369095    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.375404 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.375377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5176e098f3446ac4082c474d517eb13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal\" (UID: \"c5176e098f3446ac4082c474d517eb13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.375485 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.375410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5176e098f3446ac4082c474d517eb13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal\" (UID: \"c5176e098f3446ac4082c474d517eb13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.375485 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.375428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bbd969cc419420da6c0edfe564045bc-config\") pod \"kube-apiserver-proxy-ip-10-0-138-150.ec2.internal\" (UID: \"5bbd969cc419420da6c0edfe564045bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.469660 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.469574    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.475895 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.475869    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bbd969cc419420da6c0edfe564045bc-config\") pod \"kube-apiserver-proxy-ip-10-0-138-150.ec2.internal\" (UID: \"5bbd969cc419420da6c0edfe564045bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.475966 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.475900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5176e098f3446ac4082c474d517eb13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal\" (UID: \"c5176e098f3446ac4082c474d517eb13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.475966 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.475918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5176e098f3446ac4082c474d517eb13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal\" (UID: \"c5176e098f3446ac4082c474d517eb13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.476045 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.475979    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/5bbd969cc419420da6c0edfe564045bc-config\") pod \"kube-apiserver-proxy-ip-10-0-138-150.ec2.internal\" (UID: \"5bbd969cc419420da6c0edfe564045bc\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.476045 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.475988    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c5176e098f3446ac4082c474d517eb13-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal\" (UID: \"c5176e098f3446ac4082c474d517eb13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.476045 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.475983    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c5176e098f3446ac4082c474d517eb13-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal\" (UID: \"c5176e098f3446ac4082c474d517eb13\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.569975 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.569947    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.639276 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.639255    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.642015 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.641990    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:04.670866 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.670822    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.771281 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.771205    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.871702 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.871639    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:04.915985 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.915957    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 22:58:04.916599 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:04.916140    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 22:58:04.972192 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:04.972161    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-150.ec2.internal\" not found"
Sep 10 22:58:05.040458 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.040387    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:58:05.069267 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.069221    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 22:53:03 +0000 UTC" deadline="2028-05-04 23:38:10.914242888 +0000 UTC"
Sep 10 22:58:05.069267 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.069257    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14448h40m5.844989124s"
Sep 10 22:58:05.072392 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.072368    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 22:58:05.072591 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.072565    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:05.087480 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.087452    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 22:58:05.093337 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.093310    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 22:58:05.095177 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.095150    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal"
Sep 10 22:58:05.107296 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.107272    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 22:58:05.120520 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.120493    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4wt2c"
Sep 10 22:58:05.134378 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.134350    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4wt2c"
Sep 10 22:58:05.167976 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.167947    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:58:05.180952 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:05.180920    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5176e098f3446ac4082c474d517eb13.slice/crio-35ea2c0f3dfa031551cefdc5e776149788e56b90dc41525a53789e2106b458a6 WatchSource:0}: Error finding container 35ea2c0f3dfa031551cefdc5e776149788e56b90dc41525a53789e2106b458a6: Status 404 returned error can't find the container with id 35ea2c0f3dfa031551cefdc5e776149788e56b90dc41525a53789e2106b458a6
Sep 10 22:58:05.181155 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:05.181134    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bbd969cc419420da6c0edfe564045bc.slice/crio-1d34820959fd3bd2ad024d0c14961363d7fa4dc5ac635090bc27db95f02ec9ae WatchSource:0}: Error finding container 1d34820959fd3bd2ad024d0c14961363d7fa4dc5ac635090bc27db95f02ec9ae: Status 404 returned error can't find the container with id 1d34820959fd3bd2ad024d0c14961363d7fa4dc5ac635090bc27db95f02ec9ae
Sep 10 22:58:05.186302 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.186283    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 22:58:05.211677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.211629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal" event={"ID":"5bbd969cc419420da6c0edfe564045bc","Type":"ContainerStarted","Data":"1d34820959fd3bd2ad024d0c14961363d7fa4dc5ac635090bc27db95f02ec9ae"}
Sep 10 22:58:05.212664 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.212642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal" event={"ID":"c5176e098f3446ac4082c474d517eb13","Type":"ContainerStarted","Data":"35ea2c0f3dfa031551cefdc5e776149788e56b90dc41525a53789e2106b458a6"}
Sep 10 22:58:05.337214 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.337136    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:58:05.957360 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.957328    2577 apiserver.go:52] "Watching apiserver"
Sep 10 22:58:05.966758 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.965943    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 22:58:05.966758 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.966386    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-zd4wd","kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal","openshift-cluster-node-tuning-operator/tuned-9rt9s","openshift-dns/node-resolver-jh4r7","openshift-multus/multus-additional-cni-plugins-t6gr9","openshift-multus/network-metrics-daemon-fs2ps","openshift-network-diagnostics/network-check-target-dxcff","openshift-network-operator/iptables-alerter-t4455","kube-system/konnectivity-agent-88mdc","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n","openshift-image-registry/node-ca-9hrxg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal","openshift-multus/multus-xdq54","openshift-ovn-kubernetes/ovnkube-node-cmsj5"]
Sep 10 22:58:05.967905 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.967874    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:05.968044 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:05.967957    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:05.969076 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.969034    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.970754 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.970350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.971468 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.971383    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 22:58:05.971566 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.971484    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 22:58:05.971627 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.971601    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 22:58:05.971684 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.971647    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-2fl7x\""
Sep 10 22:58:05.972164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.971856    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:05.972164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.971909    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 22:58:05.972449 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.972411    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vsqtb\""
Sep 10 22:58:05.972449 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.972413    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:58:05.972704 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.972684    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 22:58:05.973320 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.973300    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.974248 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.974231    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 22:58:05.974371 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.974355    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 22:58:05.974443 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.974423    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-7mrt8\""
Sep 10 22:58:05.974652 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.974636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:05.974997 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:05.974710    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:05.975350 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.975333    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 22:58:05.975566 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.975546    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4lz8c\""
Sep 10 22:58:05.975760 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.975742    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 22:58:05.976136 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.976117    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:05.977476 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.977456    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:05.977563 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:05.977525    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:05.978281 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.978257    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-c5h8h\""
Sep 10 22:58:05.978476 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.978459    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 22:58:05.978476 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.978470    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 22:58:05.978978 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.978959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:05.980464 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.980443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:05.981472 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.981453    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 22:58:05.981472 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.981464    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 22:58:05.981706 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.981688    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s2kkf\""
Sep 10 22:58:05.981875 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.981857    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 22:58:05.981950 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.981921    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 22:58:05.981950 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.981942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:05.982509 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.982490    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 22:58:05.982589 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.982494    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 22:58:05.983639 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.983621    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 22:58:05.983724 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.983672    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:05.984079 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984061    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 22:58:05.984289 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-system-cni-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.984377 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984350    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 22:58:05.984616 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984583    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-c6r7l\""
Sep 10 22:58:05.984838 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984294    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a80bd00-1da2-4c71-b7cd-d26b30de5420-tmp\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.984938 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cebf87f-515b-4607-8e4e-1d53ed85cfda-tmp-dir\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:05.984938 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984893    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15bfd45a-9988-4061-9ece-ce126ab8898b-cni-binary-copy\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.984938 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984911    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 22:58:05.984938 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-cni-multus\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.985139 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984955    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-etc-kubernetes\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.985139 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.984998    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mk7hb\" (UniqueName: \"kubernetes.io/projected/15bfd45a-9988-4061-9ece-ce126ab8898b-kube-api-access-mk7hb\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.985139 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-msdgz\" (UniqueName: \"kubernetes.io/projected/5cebf87f-515b-4607-8e4e-1d53ed85cfda-kube-api-access-msdgz\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:05.985139 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-system-cni-dir\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.985139 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985121    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.985365 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dbfr7\""
Sep 10 22:58:05.985365 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985359    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 22:58:05.985579 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-systemd\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.985659 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985623    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-lib-modules\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.985659 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-conf-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.985758 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-daemon-config\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.985758 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-multus-certs\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.985929 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-tuned\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.985929 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985797    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b47dl\" (UniqueName: \"kubernetes.io/projected/7a80bd00-1da2-4c71-b7cd-d26b30de5420-kube-api-access-b47dl\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.985929 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.985929 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-netns\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.985929 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysconfig\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.985960    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986001    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmlzn\" (UniqueName: \"kubernetes.io/projected/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-kube-api-access-pmlzn\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qx8p\" (UniqueName: \"kubernetes.io/projected/875ba053-4d40-4cb4-b311-39c5ea4a56c8-kube-api-access-6qx8p\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986077    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986084    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-cni-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-k8s-cni-cncf-io\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.986164 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysctl-conf\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986168    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-var-lib-kubelet\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-host\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986218    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-run\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-cnibin\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986266    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-socket-dir-parent\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-cni-bin\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986316    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cebf87f-515b-4607-8e4e-1d53ed85cfda-hosts-file\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-kubelet\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986455    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-hostroot\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986504    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-modprobe-d\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.986574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986540    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.987084 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986603    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:05.987084 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986634    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-kubernetes\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.987084 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysctl-d\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.987084 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-sys\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:05.987084 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.986730    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cnibin\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:05.987313 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.987232    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7w5h7\""
Sep 10 22:58:05.987313 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.987284    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 22:58:05.988122 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.988098    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:58:05.988322 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:05.988305    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 22:58:06.074964 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.074924    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 22:58:06.087679 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087613    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cebf87f-515b-4607-8e4e-1d53ed85cfda-tmp-dir\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:06.087679 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-systemd\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.087679 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovnkube-config\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087705    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-socket-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72e93762-0f45-4124-a061-1b0d59a727a0-host-slash\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087811    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mk7hb\" (UniqueName: \"kubernetes.io/projected/15bfd45a-9988-4061-9ece-ce126ab8898b-kube-api-access-mk7hb\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087840    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-msdgz\" (UniqueName: \"kubernetes.io/projected/5cebf87f-515b-4607-8e4e-1d53ed85cfda-kube-api-access-msdgz\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087887    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-slash\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087911    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8d134860-6052-465b-92cd-ed4d065db2eb-konnectivity-ca\") pod \"konnectivity-agent-88mdc\" (UID: \"8d134860-6052-465b-92cd-ed4d065db2eb\") " pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087940    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-conf-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.087967 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087963    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-multus-certs\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.087987    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-tuned\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088001    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5cebf87f-515b-4607-8e4e-1d53ed85cfda-tmp-dir\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b47dl\" (UniqueName: \"kubernetes.io/projected/7a80bd00-1da2-4c71-b7cd-d26b30de5420-kube-api-access-b47dl\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qx8p\" (UniqueName: \"kubernetes.io/projected/875ba053-4d40-4cb4-b311-39c5ea4a56c8-kube-api-access-6qx8p\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088149    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-var-lib-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088177    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zd2tm\" (UniqueName: \"kubernetes.io/projected/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-kube-api-access-zd2tm\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088185    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-tuning-conf-dir\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088200    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddf1e8ab-c897-4333-8140-78d9d49493bc-host\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088221    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-multus-certs\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088225    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35acf209-c21a-4958-a3e9-0e951197405b-kubelet-config\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088248    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-conf-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088251    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-cni-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-k8s-cni-cncf-io\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.088386 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088308    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysctl-conf\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-etc-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovn-node-metrics-cert\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088493    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kprb\" (UniqueName: \"kubernetes.io/projected/72e93762-0f45-4124-a061-1b0d59a727a0-kube-api-access-2kprb\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088511    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysctl-conf\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-run\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088576    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-k8s-cni-cncf-io\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-cni-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088662    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-run\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088819    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cebf87f-515b-4607-8e4e-1d53ed85cfda-hosts-file\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-cni-bin\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088874    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovnkube-script-lib\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-registration-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088965    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-hostroot\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088993    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/5cebf87f-515b-4607-8e4e-1d53ed85cfda-hosts-file\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.088996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-modprobe-d\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.089301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089017    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-hostroot\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089059    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-run-netns\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089083    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddf1e8ab-c897-4333-8140-78d9d49493bc-serviceca\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-modprobe-d\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysctl-d\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089131    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cnibin\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089154    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-cni-netd\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089176    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-env-overrides\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cnibin\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35acf209-c21a-4958-a3e9-0e951197405b-dbus\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089257    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysctl-d\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089279    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089319    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-system-cni-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089387    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-node-log\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089427    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8d134860-6052-465b-92cd-ed4d065db2eb-agent-certs\") pod \"konnectivity-agent-88mdc\" (UID: \"8d134860-6052-465b-92cd-ed4d065db2eb\") " pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:06.090149 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15bfd45a-9988-4061-9ece-ce126ab8898b-cni-binary-copy\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089471    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-system-cni-dir\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089498    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-cni-multus\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-etc-kubernetes\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089537    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-cni-multus\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-etc-kubernetes\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-system-cni-dir\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-system-cni-dir\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089607    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-device-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-sys-fs\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-systemd\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-lib-modules\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089665    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-daemon-config\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmlzn\" (UniqueName: \"kubernetes.io/projected/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-kube-api-access-pmlzn\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-systemd\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-cni-binary-copy\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.090959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-ovn\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089795    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-netns\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089816    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-lib-modules\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089822    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysconfig\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089870    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-sysconfig\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089877    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089905    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-run-netns\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089908    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-kubelet\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-systemd-units\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.089966    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-var-lib-kubelet\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-var-lib-kubelet\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-host\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090063    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/15bfd45a-9988-4061-9ece-ce126ab8898b-cni-binary-copy\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090082    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4s9p6\" (UniqueName: \"kubernetes.io/projected/ddf1e8ab-c897-4333-8140-78d9d49493bc-kube-api-access-4s9p6\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-host\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-cnibin\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.091716 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090191    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-socket-dir-parent\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-daemon-config\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-cni-bin\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090241    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72e93762-0f45-4124-a061-1b0d59a727a0-iptables-alerter-script\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090268    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-kubelet\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-cni-bin\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090283    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-multus-socket-dir-parent\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090292    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090288    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-cnibin\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/15bfd45a-9988-4061-9ece-ce126ab8898b-host-var-lib-kubelet\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090344    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2phwj\" (UniqueName: \"kubernetes.io/projected/80f5b44c-e452-47e3-a48a-fdd89987779c-kube-api-access-2phwj\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090368    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-kubernetes\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.090369    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-sys\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090396    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090410    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-log-socket\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.094279 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.090432    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:06.590412667 +0000 UTC m=+3.026467435 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:06.095064 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090461    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-sys\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.095064 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090470    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-kubernetes\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.095064 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090508    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.095064 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.090534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a80bd00-1da2-4c71-b7cd-d26b30de5420-tmp\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.095064 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.092843    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/7a80bd00-1da2-4c71-b7cd-d26b30de5420-etc-tuned\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.095064 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.093761    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7a80bd00-1da2-4c71-b7cd-d26b30de5420-tmp\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.098352 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.098326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b47dl\" (UniqueName: \"kubernetes.io/projected/7a80bd00-1da2-4c71-b7cd-d26b30de5420-kube-api-access-b47dl\") pod \"tuned-9rt9s\" (UID: \"7a80bd00-1da2-4c71-b7cd-d26b30de5420\") " pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.098879 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.098855    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mk7hb\" (UniqueName: \"kubernetes.io/projected/15bfd45a-9988-4061-9ece-ce126ab8898b-kube-api-access-mk7hb\") pod \"multus-xdq54\" (UID: \"15bfd45a-9988-4061-9ece-ce126ab8898b\") " pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.101110 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.101074    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qx8p\" (UniqueName: \"kubernetes.io/projected/875ba053-4d40-4cb4-b311-39c5ea4a56c8-kube-api-access-6qx8p\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:06.101406 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.101384    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:58:06.101493 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.101445    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:58:06.101493 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.101461    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sj599 for pod openshift-network-diagnostics/network-check-target-dxcff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:06.101590 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.101542    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599 podName:5cf44708-d6fc-4ec3-9649-400dbed4d913 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:06.601523 +0000 UTC m=+3.037577771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-sj599" (UniqueName: "kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599") pod "network-check-target-dxcff" (UID: "5cf44708-d6fc-4ec3-9649-400dbed4d913") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:06.103977 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.103927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-msdgz\" (UniqueName: \"kubernetes.io/projected/5cebf87f-515b-4607-8e4e-1d53ed85cfda-kube-api-access-msdgz\") pod \"node-resolver-jh4r7\" (UID: \"5cebf87f-515b-4607-8e4e-1d53ed85cfda\") " pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:06.106445 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.106422    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmlzn\" (UniqueName: \"kubernetes.io/projected/8a03f169-6f4e-4c71-aedd-aa79c1e9e859-kube-api-access-pmlzn\") pod \"multus-additional-cni-plugins-t6gr9\" (UID: \"8a03f169-6f4e-4c71-aedd-aa79c1e9e859\") " pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.136340 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.136231    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:53:05 +0000 UTC" deadline="2028-02-10 06:10:38.402226613 +0000 UTC"
Sep 10 22:58:06.136340 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.136306    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12415h12m32.26592515s"
Sep 10 22:58:06.155376 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.155349    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 22:58:06.191851 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.191814    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.191851 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.191856    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4s9p6\" (UniqueName: \"kubernetes.io/projected/ddf1e8ab-c897-4333-8140-78d9d49493bc-kube-api-access-4s9p6\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.192086 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.191875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72e93762-0f45-4124-a061-1b0d59a727a0-iptables-alerter-script\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.192086 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.191946    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192086 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.191967    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-etc-selinux\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.192086 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.191983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2phwj\" (UniqueName: \"kubernetes.io/projected/80f5b44c-e452-47e3-a48a-fdd89987779c-kube-api-access-2phwj\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.192086 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192032    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-log-socket\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192086 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192050    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192086 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192093    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-systemd\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-run-ovn-kubernetes\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovnkube-config\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192132    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-log-socket\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192143    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-socket-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.192341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192168    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72e93762-0f45-4124-a061-1b0d59a727a0-host-slash\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.192619 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-socket-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.192674 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-slash\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192747 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192693    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8d134860-6052-465b-92cd-ed4d065db2eb-konnectivity-ca\") pod \"konnectivity-agent-88mdc\" (UID: \"8d134860-6052-465b-92cd-ed4d065db2eb\") " pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:06.192816 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192750    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-var-lib-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192816 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192798    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zd2tm\" (UniqueName: \"kubernetes.io/projected/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-kube-api-access-zd2tm\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.192917 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192844    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddf1e8ab-c897-4333-8140-78d9d49493bc-host\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.192917 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192887    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35acf209-c21a-4958-a3e9-0e951197405b-kubelet-config\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.192917 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192900    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/72e93762-0f45-4124-a061-1b0d59a727a0-iptables-alerter-script\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.193066 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192916    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-etc-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193066 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-systemd\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193066 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.192963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-etc-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193066 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/35acf209-c21a-4958-a3e9-0e951197405b-kubelet-config\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.193246 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193173    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovn-node-metrics-cert\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193436 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ddf1e8ab-c897-4333-8140-78d9d49493bc-host\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.193436 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-slash\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193436 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/72e93762-0f45-4124-a061-1b0d59a727a0-host-slash\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.193436 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-var-lib-openvswitch\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kprb\" (UniqueName: \"kubernetes.io/projected/72e93762-0f45-4124-a061-1b0d59a727a0-kube-api-access-2kprb\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.193710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193501    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovnkube-config\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/8d134860-6052-465b-92cd-ed4d065db2eb-konnectivity-ca\") pod \"konnectivity-agent-88mdc\" (UID: \"8d134860-6052-465b-92cd-ed4d065db2eb\") " pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:06.193710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193580    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-cni-bin\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.193710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovnkube-script-lib\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194034 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193849    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-cni-bin\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194034 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194034 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.193953    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-registration-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.194034 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-run-netns\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194034 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194028    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddf1e8ab-c897-4333-8140-78d9d49493bc-serviceca\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194061    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-cni-netd\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194085    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-registration-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-env-overrides\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194171    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35acf209-c21a-4958-a3e9-0e951197405b-dbus\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194232    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-node-log\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8d134860-6052-465b-92cd-ed4d065db2eb-agent-certs\") pod \"konnectivity-agent-88mdc\" (UID: \"8d134860-6052-465b-92cd-ed4d065db2eb\") " pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194270    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovnkube-script-lib\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194317 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194290    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-device-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-sys-fs\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194359    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-run-netns\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194362    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-ovn\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194410    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-kubelet-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-kubelet\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194451    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-run-ovn\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194455    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-systemd-units\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194497    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-env-overrides\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/35acf209-c21a-4958-a3e9-0e951197405b-dbus\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194541    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-systemd-units\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-cni-netd\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-host-kubelet\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-device-dir\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.194669    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:06.194821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.194755    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/80f5b44c-e452-47e3-a48a-fdd89987779c-sys-fs\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.195495 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.194894    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret podName:35acf209-c21a-4958-a3e9-0e951197405b nodeName:}" failed. No retries permitted until 2026-09-10 22:58:06.69486575 +0000 UTC m=+3.130920521 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret") pod "global-pull-secret-syncer-zd4wd" (UID: "35acf209-c21a-4958-a3e9-0e951197405b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:06.195495 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.195231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-node-log\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.197326 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.196303    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ddf1e8ab-c897-4333-8140-78d9d49493bc-serviceca\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.198648 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.198620    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/8d134860-6052-465b-92cd-ed4d065db2eb-agent-certs\") pod \"konnectivity-agent-88mdc\" (UID: \"8d134860-6052-465b-92cd-ed4d065db2eb\") " pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:06.199162 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.199140    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-ovn-node-metrics-cert\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.202390 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.202361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2phwj\" (UniqueName: \"kubernetes.io/projected/80f5b44c-e452-47e3-a48a-fdd89987779c-kube-api-access-2phwj\") pod \"aws-ebs-csi-driver-node-tj77n\" (UID: \"80f5b44c-e452-47e3-a48a-fdd89987779c\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.202995 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.202969    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zd2tm\" (UniqueName: \"kubernetes.io/projected/ca19b165-c732-475a-af7b-f5c1f9c3e4fe-kube-api-access-zd2tm\") pod \"ovnkube-node-cmsj5\" (UID: \"ca19b165-c732-475a-af7b-f5c1f9c3e4fe\") " pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.203499 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.203475    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kprb\" (UniqueName: \"kubernetes.io/projected/72e93762-0f45-4124-a061-1b0d59a727a0-kube-api-access-2kprb\") pod \"iptables-alerter-t4455\" (UID: \"72e93762-0f45-4124-a061-1b0d59a727a0\") " pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.203661 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.203640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4s9p6\" (UniqueName: \"kubernetes.io/projected/ddf1e8ab-c897-4333-8140-78d9d49493bc-kube-api-access-4s9p6\") pod \"node-ca-9hrxg\" (UID: \"ddf1e8ab-c897-4333-8140-78d9d49493bc\") " pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.286639 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.286544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-xdq54"
Sep 10 22:58:06.293503 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.293472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-9rt9s"
Sep 10 22:58:06.303285 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.303248    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jh4r7"
Sep 10 22:58:06.311205 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.311178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-t6gr9"
Sep 10 22:58:06.320043 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.320008    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:06.327112 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.327086    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:06.339833 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.339797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n"
Sep 10 22:58:06.346735 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.346705    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9hrxg"
Sep 10 22:58:06.352585 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.352559    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t4455"
Sep 10 22:58:06.597137 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.597034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:06.597363 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.597165    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:06.597363 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.597230    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:07.597215228 +0000 UTC m=+4.033269979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:06.697606 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.697564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:06.697773 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:06.697660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:06.697773 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.697727    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:06.697773 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.697769    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:58:06.697967 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.697802    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:58:06.697967 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.697815    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret podName:35acf209-c21a-4958-a3e9-0e951197405b nodeName:}" failed. No retries permitted until 2026-09-10 22:58:07.6977932 +0000 UTC m=+4.133847966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret") pod "global-pull-secret-syncer-zd4wd" (UID: "35acf209-c21a-4958-a3e9-0e951197405b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:06.697967 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.697817    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sj599 for pod openshift-network-diagnostics/network-check-target-dxcff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:06.697967 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:06.697858    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599 podName:5cf44708-d6fc-4ec3-9649-400dbed4d913 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:07.697848966 +0000 UTC m=+4.133903717 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj599" (UniqueName: "kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599") pod "network-check-target-dxcff" (UID: "5cf44708-d6fc-4ec3-9649-400dbed4d913") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:07.137264 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.137209    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 22:53:05 +0000 UTC" deadline="2028-03-07 19:30:26.02004215 +0000 UTC"
Sep 10 22:58:07.137264 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.137257    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13052h32m18.882787597s"
Sep 10 22:58:07.165926 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:07.165743    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cebf87f_515b_4607_8e4e_1d53ed85cfda.slice/crio-a844f2d0a0a1b2f32b84ec7a8804615eeccce2afaa86f5239c28d13ce0b4b2a6 WatchSource:0}: Error finding container a844f2d0a0a1b2f32b84ec7a8804615eeccce2afaa86f5239c28d13ce0b4b2a6: Status 404 returned error can't find the container with id a844f2d0a0a1b2f32b84ec7a8804615eeccce2afaa86f5239c28d13ce0b4b2a6
Sep 10 22:58:07.166111 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:07.166089    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15bfd45a_9988_4061_9ece_ce126ab8898b.slice/crio-73081d9f0a849c09074147ea3a9db23f92b3734d5165e0cc65ab8b56e2aa326a WatchSource:0}: Error finding container 73081d9f0a849c09074147ea3a9db23f92b3734d5165e0cc65ab8b56e2aa326a: Status 404 returned error can't find the container with id 73081d9f0a849c09074147ea3a9db23f92b3734d5165e0cc65ab8b56e2aa326a
Sep 10 22:58:07.167627 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:07.167585    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72e93762_0f45_4124_a061_1b0d59a727a0.slice/crio-ddabc529e59ba40ffa7e84554d30f689e25029f56bdc51799b02e3e634e4b209 WatchSource:0}: Error finding container ddabc529e59ba40ffa7e84554d30f689e25029f56bdc51799b02e3e634e4b209: Status 404 returned error can't find the container with id ddabc529e59ba40ffa7e84554d30f689e25029f56bdc51799b02e3e634e4b209
Sep 10 22:58:07.173577 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:07.173551    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a80bd00_1da2_4c71_b7cd_d26b30de5420.slice/crio-fa9101e7ee2eb5e4c3ce4dea066c296cdfa726c366a4b33e324419a4a62a505a WatchSource:0}: Error finding container fa9101e7ee2eb5e4c3ce4dea066c296cdfa726c366a4b33e324419a4a62a505a: Status 404 returned error can't find the container with id fa9101e7ee2eb5e4c3ce4dea066c296cdfa726c366a4b33e324419a4a62a505a
Sep 10 22:58:07.176690 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:07.176660    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddf1e8ab_c897_4333_8140_78d9d49493bc.slice/crio-29da8d080c1a71d36633855c73bb850ae4c496bdb3735447139a137871d9c40c WatchSource:0}: Error finding container 29da8d080c1a71d36633855c73bb850ae4c496bdb3735447139a137871d9c40c: Status 404 returned error can't find the container with id 29da8d080c1a71d36633855c73bb850ae4c496bdb3735447139a137871d9c40c
Sep 10 22:58:07.177380 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:07.177357    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a03f169_6f4e_4c71_aedd_aa79c1e9e859.slice/crio-0fe3d7f2368225ea96e0c1bace7f04b41025e2dba38013971acd309bc3aa55df WatchSource:0}: Error finding container 0fe3d7f2368225ea96e0c1bace7f04b41025e2dba38013971acd309bc3aa55df: Status 404 returned error can't find the container with id 0fe3d7f2368225ea96e0c1bace7f04b41025e2dba38013971acd309bc3aa55df
Sep 10 22:58:07.179994 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:07.179171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d134860_6052_465b_92cd_ed4d065db2eb.slice/crio-77fe91dc89c8ac64bdf36607e599bf3d3afdbe2cf480b4cb18e46070e5d056f0 WatchSource:0}: Error finding container 77fe91dc89c8ac64bdf36607e599bf3d3afdbe2cf480b4cb18e46070e5d056f0: Status 404 returned error can't find the container with id 77fe91dc89c8ac64bdf36607e599bf3d3afdbe2cf480b4cb18e46070e5d056f0
Sep 10 22:58:07.208257 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.208224    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:07.208417 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.208332    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:07.215995 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.215964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9rt9s" event={"ID":"7a80bd00-1da2-4c71-b7cd-d26b30de5420","Type":"ContainerStarted","Data":"fa9101e7ee2eb5e4c3ce4dea066c296cdfa726c366a4b33e324419a4a62a505a"}
Sep 10 22:58:07.218669 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.218637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88mdc" event={"ID":"8d134860-6052-465b-92cd-ed4d065db2eb","Type":"ContainerStarted","Data":"77fe91dc89c8ac64bdf36607e599bf3d3afdbe2cf480b4cb18e46070e5d056f0"}
Sep 10 22:58:07.221156 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.221121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"8287c6ee4e74139942ce8dbd57fc114b06b8798ed818ddcd6a774ecbc68c72ef"}
Sep 10 22:58:07.222460 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.222438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xdq54" event={"ID":"15bfd45a-9988-4061-9ece-ce126ab8898b","Type":"ContainerStarted","Data":"73081d9f0a849c09074147ea3a9db23f92b3734d5165e0cc65ab8b56e2aa326a"}
Sep 10 22:58:07.223886 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.223630    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n" event={"ID":"80f5b44c-e452-47e3-a48a-fdd89987779c","Type":"ContainerStarted","Data":"8a4a0086e013855320c2468b644d4895ad4ee21bfe84bb9aa8fee65201b28e38"}
Sep 10 22:58:07.224966 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.224941    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t4455" event={"ID":"72e93762-0f45-4124-a061-1b0d59a727a0","Type":"ContainerStarted","Data":"ddabc529e59ba40ffa7e84554d30f689e25029f56bdc51799b02e3e634e4b209"}
Sep 10 22:58:07.226106 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.226081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jh4r7" event={"ID":"5cebf87f-515b-4607-8e4e-1d53ed85cfda","Type":"ContainerStarted","Data":"a844f2d0a0a1b2f32b84ec7a8804615eeccce2afaa86f5239c28d13ce0b4b2a6"}
Sep 10 22:58:07.227286 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.227265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hrxg" event={"ID":"ddf1e8ab-c897-4333-8140-78d9d49493bc","Type":"ContainerStarted","Data":"29da8d080c1a71d36633855c73bb850ae4c496bdb3735447139a137871d9c40c"}
Sep 10 22:58:07.228481 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.228440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerStarted","Data":"0fe3d7f2368225ea96e0c1bace7f04b41025e2dba38013971acd309bc3aa55df"}
Sep 10 22:58:07.604395 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.603886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:07.604395 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.604141    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:07.604395 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.604215    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:09.604195428 +0000 UTC m=+6.040250183 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:07.705694 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.705594    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:07.705897 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:07.705704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:07.705897 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.705866    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:58:07.705897 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.705887    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:58:07.705897 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.705900    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sj599 for pod openshift-network-diagnostics/network-check-target-dxcff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:07.706103 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.705984    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599 podName:5cf44708-d6fc-4ec3-9649-400dbed4d913 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:09.70596467 +0000 UTC m=+6.142019423 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj599" (UniqueName: "kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599") pod "network-check-target-dxcff" (UID: "5cf44708-d6fc-4ec3-9649-400dbed4d913") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:07.706762 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.706414    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:07.706762 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:07.706566    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret podName:35acf209-c21a-4958-a3e9-0e951197405b nodeName:}" failed. No retries permitted until 2026-09-10 22:58:09.70654974 +0000 UTC m=+6.142604497 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret") pod "global-pull-secret-syncer-zd4wd" (UID: "35acf209-c21a-4958-a3e9-0e951197405b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:08.210846 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:08.210232    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:08.210846 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:08.210375    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:08.211844 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:08.211627    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:08.211844 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:08.211764    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:08.246798 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:08.246047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal" event={"ID":"5bbd969cc419420da6c0edfe564045bc","Type":"ContainerStarted","Data":"275d68353873169828b89b19f9d5566c96757836309d9ca5c41cebe1a218a602"}
Sep 10 22:58:08.256930 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:08.256890    2577 generic.go:356] "Generic (PLEG): container finished" podID="c5176e098f3446ac4082c474d517eb13" containerID="888c210e0f19172249983ba76970c41ef7b9dc517dbcd9a94994e21c745c5a22" exitCode=0
Sep 10 22:58:08.257092 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:08.256953    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal" event={"ID":"c5176e098f3446ac4082c474d517eb13","Type":"ContainerDied","Data":"888c210e0f19172249983ba76970c41ef7b9dc517dbcd9a94994e21c745c5a22"}
Sep 10 22:58:08.267449 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:08.263795    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-150.ec2.internal" podStartSLOduration=3.263759441 podStartE2EDuration="3.263759441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:08.26280981 +0000 UTC m=+4.698864579" watchObservedRunningTime="2026-09-10 22:58:08.263759441 +0000 UTC m=+4.699814215"
Sep 10 22:58:09.208156 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:09.208119    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:09.208360 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.208262    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:09.281959 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:09.281918    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal" event={"ID":"c5176e098f3446ac4082c474d517eb13","Type":"ContainerStarted","Data":"e86fc1b989d72d18ab74099411caa37458a08e8d24c61f7b8467a5ffdfcd6588"}
Sep 10 22:58:09.307561 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:09.307109    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-150.ec2.internal" podStartSLOduration=4.307086932 podStartE2EDuration="4.307086932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:09.307059039 +0000 UTC m=+5.743113812" watchObservedRunningTime="2026-09-10 22:58:09.307086932 +0000 UTC m=+5.743141708"
Sep 10 22:58:09.624250 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:09.623464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:09.624250 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.623664    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:09.624250 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.623727    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:13.623709844 +0000 UTC m=+10.059764602 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:09.724802    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:09.724900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.725093    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.725114    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.725131    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sj599 for pod openshift-network-diagnostics/network-check-target-dxcff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.725189    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599 podName:5cf44708-d6fc-4ec3-9649-400dbed4d913 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:13.725171693 +0000 UTC m=+10.161226449 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj599" (UniqueName: "kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599") pod "network-check-target-dxcff" (UID: "5cf44708-d6fc-4ec3-9649-400dbed4d913") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.725753    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:09.725877 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:09.725838    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret podName:35acf209-c21a-4958-a3e9-0e951197405b nodeName:}" failed. No retries permitted until 2026-09-10 22:58:13.725823867 +0000 UTC m=+10.161878621 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret") pod "global-pull-secret-syncer-zd4wd" (UID: "35acf209-c21a-4958-a3e9-0e951197405b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:10.209747 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:10.209708    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:10.209953 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:10.209862    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:10.210021 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:10.209994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:10.210124 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:10.210103    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:11.208759 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:11.208725    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:11.209244 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:11.208885    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:12.209205 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:12.209164    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:12.209672 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:12.209308    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:12.209809 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:12.209774    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:12.209919 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:12.209896    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:13.208771 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:13.208729    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:13.208968 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.208864    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:13.662684 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:13.662592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:13.663125 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.662763    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:13.663125 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.662855    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:21.662834305 +0000 UTC m=+18.098889059 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:13.763474 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:13.763428    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:13.763655 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:13.763519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:13.763655 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.763596    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:13.763655 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.763648    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:58:13.763766 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.763665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret podName:35acf209-c21a-4958-a3e9-0e951197405b nodeName:}" failed. No retries permitted until 2026-09-10 22:58:21.763645087 +0000 UTC m=+18.199699850 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret") pod "global-pull-secret-syncer-zd4wd" (UID: "35acf209-c21a-4958-a3e9-0e951197405b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:13.763766 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.763675    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:58:13.763766 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.763689    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sj599 for pod openshift-network-diagnostics/network-check-target-dxcff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:13.763766 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:13.763738    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599 podName:5cf44708-d6fc-4ec3-9649-400dbed4d913 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:21.76372103 +0000 UTC m=+18.199775784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj599" (UniqueName: "kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599") pod "network-check-target-dxcff" (UID: "5cf44708-d6fc-4ec3-9649-400dbed4d913") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:14.209976 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:14.209662    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:14.209976 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:14.209808    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:14.209976 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:14.209921    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:14.209976 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:14.209815    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:15.208298 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:15.208256    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:15.208702 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:15.208426    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:16.209068 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:16.209036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:16.209549 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:16.209144    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:16.209549 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:16.209197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:16.209549 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:16.209317    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:17.208723 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:17.208687    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:17.208941 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:17.208820    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:18.208348 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:18.208313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:18.208829 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:18.208313    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:18.208829 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:18.208439    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:18.208829 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:18.208519    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:19.208434 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:19.208395    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:19.208929 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:19.208572    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:20.208306 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:20.208266    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:20.208505 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:20.208271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:20.208505 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:20.208409    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:20.208901 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:20.208495    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:21.209096 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:21.209059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:21.209594 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.209178    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:21.727053 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:21.727018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:21.727206 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.727146    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:21.727281 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.727211    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:37.727190241 +0000 UTC m=+34.163244993 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:21.828325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:21.828284    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:21.828492 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:21.828344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:21.828492 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.828480    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:58:21.828590 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.828514    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:58:21.828590 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.828529    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sj599 for pod openshift-network-diagnostics/network-check-target-dxcff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:21.828590 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.828482    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:21.828590 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.828589    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599 podName:5cf44708-d6fc-4ec3-9649-400dbed4d913 nodeName:}" failed. No retries permitted until 2026-09-10 22:58:37.828568865 +0000 UTC m=+34.264623619 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj599" (UniqueName: "kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599") pod "network-check-target-dxcff" (UID: "5cf44708-d6fc-4ec3-9649-400dbed4d913") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:21.828759 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:21.828635    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret podName:35acf209-c21a-4958-a3e9-0e951197405b nodeName:}" failed. No retries permitted until 2026-09-10 22:58:37.828619232 +0000 UTC m=+34.264673982 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret") pod "global-pull-secret-syncer-zd4wd" (UID: "35acf209-c21a-4958-a3e9-0e951197405b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:22.208511 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:22.208426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:22.208682 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:22.208555    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:22.208682 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:22.208613    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:22.208805 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:22.208745    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:23.208320 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:23.208284    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:23.208810 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:23.208415    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:24.209510 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:24.209419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:24.209998 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:24.209549    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:24.209998 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:24.209610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:24.209998 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:24.209715    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:25.209239 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.209081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:25.209365 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:25.209323    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:25.315346 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.315104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jh4r7" event={"ID":"5cebf87f-515b-4607-8e4e-1d53ed85cfda","Type":"ContainerStarted","Data":"6cec25904166020e47c4b204a499644b28a671106c18736be94eafb186f8654e"}
Sep 10 22:58:25.316629 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.316601    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9hrxg" event={"ID":"ddf1e8ab-c897-4333-8140-78d9d49493bc","Type":"ContainerStarted","Data":"8e974bd0dde36db8dbc310012fbb29b2ca8588ebf0985a4a8398183029698eb7"}
Sep 10 22:58:25.318238 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.318213    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a03f169-6f4e-4c71-aedd-aa79c1e9e859" containerID="9187e49dbbde6207a060d1bb86e0e5b4ec5bea69c3581ae3355c4bb8cf4320cb" exitCode=0
Sep 10 22:58:25.318336 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.318293    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerDied","Data":"9187e49dbbde6207a060d1bb86e0e5b4ec5bea69c3581ae3355c4bb8cf4320cb"}
Sep 10 22:58:25.320353 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.320326    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-9rt9s" event={"ID":"7a80bd00-1da2-4c71-b7cd-d26b30de5420","Type":"ContainerStarted","Data":"d2b02c41a61ad554fe849ae5fab5ebe706ebfe52cd73e1a9ea91bb184f3f9f0e"}
Sep 10 22:58:25.322634 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.322605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-88mdc" event={"ID":"8d134860-6052-465b-92cd-ed4d065db2eb","Type":"ContainerStarted","Data":"d9cbd402381bc836d4ef3f6c91fa59c8906900676168357c1557fa4ba249186c"}
Sep 10 22:58:25.324473 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.324103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:25.324929 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.324912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:25.325622 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.325600    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"cbf87270f9ee4cff7cb59d784b17e7aadb1916cfaa617b398d722bab976eac33"}
Sep 10 22:58:25.325708 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.325656    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"0a11b336ccb20af34e8bb550f8fb0264006bd6a0f77824cc5d08210268469f5b"}
Sep 10 22:58:25.325708 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.325699    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"172695ded0f024c82b1336eafc577b3d019e9b2dd199078fa49adf70deb94cf7"}
Sep 10 22:58:25.330437 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.330407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-xdq54" event={"ID":"15bfd45a-9988-4061-9ece-ce126ab8898b","Type":"ContainerStarted","Data":"3c88d2aec68ecc3fa3517e4a8a4f514e9f5479fea28368ed9b258bc7f05732a6"}
Sep 10 22:58:25.332356 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.332335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n" event={"ID":"80f5b44c-e452-47e3-a48a-fdd89987779c","Type":"ContainerStarted","Data":"6d471ee7c19aa42093aabcf2c0595e878210cc340149519847bc134321f3a9fb"}
Sep 10 22:58:25.336427 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.336389    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jh4r7" podStartSLOduration=3.651857103 podStartE2EDuration="21.336376545s" totalImagesPullingTime="17.684519442s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:25.335731995 +0000 UTC m=+21.771786793" watchObservedRunningTime="2026-09-10 22:58:25.336376545 +0000 UTC m=+21.772431319"
Sep 10 22:58:25.374498 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.374448    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-88mdc" podStartSLOduration=11.061346326 podStartE2EDuration="21.37443267s" totalImagesPullingTime="10.313086344s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:25.374388313 +0000 UTC m=+21.810443087" watchObservedRunningTime="2026-09-10 22:58:25.37443267 +0000 UTC m=+21.810487444"
Sep 10 22:58:25.374644 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.374626    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-xdq54" podStartSLOduration=3.626441029 podStartE2EDuration="21.374620302s" totalImagesPullingTime="17.748179273s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:25.352136689 +0000 UTC m=+21.788191463" watchObservedRunningTime="2026-09-10 22:58:25.374620302 +0000 UTC m=+21.810675075"
Sep 10 22:58:25.390966 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.390924    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9hrxg" podStartSLOduration=3.717008344 podStartE2EDuration="21.390908237s" totalImagesPullingTime="17.673899893s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:25.390627913 +0000 UTC m=+21.826682687" watchObservedRunningTime="2026-09-10 22:58:25.390908237 +0000 UTC m=+21.826963009"
Sep 10 22:58:25.429952 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:25.429905    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-9rt9s" podStartSLOduration=3.733848009 podStartE2EDuration="21.42988877s" totalImagesPullingTime="17.696040761s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:25.408523221 +0000 UTC m=+21.844577994" watchObservedRunningTime="2026-09-10 22:58:25.42988877 +0000 UTC m=+21.865943542"
Sep 10 22:58:26.090008 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.089983    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 22:58:26.159296 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.159169    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T22:58:26.090004966Z","UUID":"efbe01df-7238-4cbe-982f-0fa3cd83d46c","Handler":null,"Name":"","Endpoint":""}
Sep 10 22:58:26.161213 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.161192    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 22:58:26.161353 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.161225    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 22:58:26.208965 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.208937    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:26.209153 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.208943    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:26.209153 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:26.209058    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:26.209153 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:26.209127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:26.338352 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.338270    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"a42709503ba1d2ba16aa3430904ebd4f056cf820b81241f1268ae5ec947d2859"}
Sep 10 22:58:26.338352 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.338347    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"6036e8a61e3512d0600ca8a4eca01eb7627f3fbc3872a129a1869811d40d53e5"}
Sep 10 22:58:26.339140 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.338391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"d81d3ba85fa66e9c012259c00d45d8b7e925e27a8809cbd45783b4a71be5e362"}
Sep 10 22:58:26.340210 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.340186    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n" event={"ID":"80f5b44c-e452-47e3-a48a-fdd89987779c","Type":"ContainerStarted","Data":"4876e7e4648ed8a4b8e3b9bf73fdd6c9651fbce48fb14c4881f338110c1d4ce3"}
Sep 10 22:58:26.341558 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.341530    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t4455" event={"ID":"72e93762-0f45-4124-a061-1b0d59a727a0","Type":"ContainerStarted","Data":"5acc77fde8ec27de71124eea89f0782d741fb156f6bde0be0bc11865f7dce220"}
Sep 10 22:58:26.342291 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.342274    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:26.342821 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.342806    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-88mdc"
Sep 10 22:58:26.359802 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:26.358967    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t4455" podStartSLOduration=4.676748235 podStartE2EDuration="22.358948849s" totalImagesPullingTime="17.682200614s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:26.358352571 +0000 UTC m=+22.794407346" watchObservedRunningTime="2026-09-10 22:58:26.358948849 +0000 UTC m=+22.795003625"
Sep 10 22:58:27.208247 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:27.208219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:27.208388 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:27.208360    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:27.347027 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:27.346989    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n" event={"ID":"80f5b44c-e452-47e3-a48a-fdd89987779c","Type":"ContainerStarted","Data":"30fda796f49d43766aedd595022ebc5edaf43520e32f5b945d55700f44234b35"}
Sep 10 22:58:27.370733 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:27.370641    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-tj77n" podStartSLOduration=3.7573085539999997 podStartE2EDuration="23.370625763s" totalImagesPullingTime="19.613317209s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:27.369659416 +0000 UTC m=+23.805714201" watchObservedRunningTime="2026-09-10 22:58:27.370625763 +0000 UTC m=+23.806680538"
Sep 10 22:58:28.208829 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:28.208796    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:28.208829 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:28.208813    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:28.209084 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:28.208923    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:28.209084 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:28.209060    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:28.354060 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:28.353821    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"3ac17800f4de1f0e21ecbd50a1e6d75483647c08805ca14594768bdd611ec42a"}
Sep 10 22:58:29.208058 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:29.208019    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:29.208249 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:29.208156    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:30.208616 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.208585    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:30.209066 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.208588    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:30.209066 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:30.208686    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:30.209066 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:30.208763    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:30.360303 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.360272    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a03f169-6f4e-4c71-aedd-aa79c1e9e859" containerID="c760124438f649dc6f9bd176d567d2e3b332e58af2e3b4d3f0dc2ee6a3c9392a" exitCode=0
Sep 10 22:58:30.360467 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.360318    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerDied","Data":"c760124438f649dc6f9bd176d567d2e3b332e58af2e3b4d3f0dc2ee6a3c9392a"}
Sep 10 22:58:30.364342 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.364308    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" event={"ID":"ca19b165-c732-475a-af7b-f5c1f9c3e4fe","Type":"ContainerStarted","Data":"50b93431752dc5842cb1181443f2b95f66c058def2582e6b433ce6a003776d75"}
Sep 10 22:58:30.364575 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.364561    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:30.364637 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.364631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:30.364694 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.364684    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:30.379911 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.379884    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:30.380040 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.380018    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:58:30.413933 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:30.413879    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5" podStartSLOduration=8.471512062 podStartE2EDuration="26.413861498s" totalImagesPullingTime="17.942349436s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:30.412402473 +0000 UTC m=+26.848457246" watchObservedRunningTime="2026-09-10 22:58:30.413861498 +0000 UTC m=+26.849916270"
Sep 10 22:58:31.208710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.208520    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:31.209090 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:31.208805    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:31.369765 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.369683    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a03f169-6f4e-4c71-aedd-aa79c1e9e859" containerID="22c9870b5f9abed9e7409893aa8266c71101f2d32004fdff41ab7c065fd108ab" exitCode=0
Sep 10 22:58:31.369765 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.369750    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerDied","Data":"22c9870b5f9abed9e7409893aa8266c71101f2d32004fdff41ab7c065fd108ab"}
Sep 10 22:58:31.836324 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.836271    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zd4wd"]
Sep 10 22:58:31.836509 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.836384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:31.836575 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:31.836502    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:31.844276 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.843858    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fs2ps"]
Sep 10 22:58:31.844276 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.844041    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:31.844276 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:31.844194    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:31.845174 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.845148    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dxcff"]
Sep 10 22:58:31.845277 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:31.845239    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:31.845349 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:31.845328    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:32.374092 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:32.374007    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a03f169-6f4e-4c71-aedd-aa79c1e9e859" containerID="e0db7b946b5a73c5bfe893d44d7f89009da190cca2e90542703e12a873418adf" exitCode=0
Sep 10 22:58:32.374092 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:32.374065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerDied","Data":"e0db7b946b5a73c5bfe893d44d7f89009da190cca2e90542703e12a873418adf"}
Sep 10 22:58:33.000412 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:33.000373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:33.130914 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:33.130878    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:33.208790 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:33.208756    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:33.208968 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:33.208806    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:33.208968 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:33.208817    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:33.208968 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:33.208905    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:33.209109 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:33.209037    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:33.209155 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:33.209121    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:33.624048 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:33.624016    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:35.209316 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:35.209104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:35.209823 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:35.209111    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:35.209823 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:35.209135    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:35.209823 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:35.209422    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:35.209823 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:35.209538    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:35.209823 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:35.209633    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:36.707774 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:36.707741    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:36.773620 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:36.773600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:37.208343 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.208301    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:37.208343 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.208338    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:37.208574 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.208324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:37.208574 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.208426    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dxcff" podUID="5cf44708-d6fc-4ec3-9649-400dbed4d913"
Sep 10 22:58:37.208574 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.208492    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-fs2ps" podUID="875ba053-4d40-4cb4-b311-39c5ea4a56c8"
Sep 10 22:58:37.208574 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.208560    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zd4wd" podUID="35acf209-c21a-4958-a3e9-0e951197405b"
Sep 10 22:58:37.389026 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.388992    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a03f169-6f4e-4c71-aedd-aa79c1e9e859" containerID="e1ec1b19c97c894bef7277ead837d14945898b0fdf132cbae132194360536beb" exitCode=0
Sep 10 22:58:37.389218 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.389059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerDied","Data":"e1ec1b19c97c894bef7277ead837d14945898b0fdf132cbae132194360536beb"}
Sep 10 22:58:37.518680 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.518651    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:37.612534 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.612505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:37.751182 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.751089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:37.751718 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.751257    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:37.751718 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.751345    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:09.751322915 +0000 UTC m=+66.187377667 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 22:58:37.851977 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.851929    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:37.852151 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:37.851995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:37.852151 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.852105    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:37.852228 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.852153    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 22:58:37.852228 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.852170    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 22:58:37.852228 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.852185    2577 projected.go:196] Error preparing data for projected volume kube-api-access-sj599 for pod openshift-network-diagnostics/network-check-target-dxcff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:37.852228 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.852172    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret podName:35acf209-c21a-4958-a3e9-0e951197405b nodeName:}" failed. No retries permitted until 2026-09-10 22:59:09.8521541 +0000 UTC m=+66.288208869 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret") pod "global-pull-secret-syncer-zd4wd" (UID: "35acf209-c21a-4958-a3e9-0e951197405b") : object "kube-system"/"original-pull-secret" not registered
Sep 10 22:58:37.852228 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:37.852227    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599 podName:5cf44708-d6fc-4ec3-9649-400dbed4d913 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:09.852216982 +0000 UTC m=+66.288271736 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-sj599" (UniqueName: "kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599") pod "network-check-target-dxcff" (UID: "5cf44708-d6fc-4ec3-9649-400dbed4d913") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 22:58:38.394751 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.394717    2577 generic.go:356] "Generic (PLEG): container finished" podID="8a03f169-6f4e-4c71-aedd-aa79c1e9e859" containerID="18d70d48ff7833480f92a12dbff161e52b3c6eb3cdd7996f41678e6b35164d4c" exitCode=0
Sep 10 22:58:38.394928 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.394759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerDied","Data":"18d70d48ff7833480f92a12dbff161e52b3c6eb3cdd7996f41678e6b35164d4c"}
Sep 10 22:58:38.883952 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.883759    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-150.ec2.internal" event="NodeReady"
Sep 10 22:58:38.884342 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.884059    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 22:58:38.943093 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.943062    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sqndg"]
Sep 10 22:58:38.968756 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.968727    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vlhs2"]
Sep 10 22:58:38.968935 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.968878    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:38.971176 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.971154    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 22:58:38.971288 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.971154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-677w4\""
Sep 10 22:58:38.971405 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.971392    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 22:58:38.992710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.992684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqndg"]
Sep 10 22:58:38.992710 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.992720    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlhs2"]
Sep 10 22:58:38.992894 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.992818    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:38.995224 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.995206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 22:58:38.995409 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.995393    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-2sqgd\""
Sep 10 22:58:38.995482 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.995426    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 22:58:38.995482 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:38.995434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 22:58:39.060999 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.060961    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-tmp-dir\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.060999 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.061013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnn7h\" (UniqueName: \"kubernetes.io/projected/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-kube-api-access-dnn7h\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.061266 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.061099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-config-volume\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.061266 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.061135    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.161539 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.161511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sxxr\" (UniqueName: \"kubernetes.io/projected/3f5cfece-be37-4c58-80b8-d099ea39e781-kube-api-access-6sxxr\") pod \"ingress-canary-vlhs2\" (UID: \"3f5cfece-be37-4c58-80b8-d099ea39e781\") " pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:39.161702 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.161562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-tmp-dir\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.161702 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.161610    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnn7h\" (UniqueName: \"kubernetes.io/projected/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-kube-api-access-dnn7h\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.161702 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.161633    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f5cfece-be37-4c58-80b8-d099ea39e781-cert\") pod \"ingress-canary-vlhs2\" (UID: \"3f5cfece-be37-4c58-80b8-d099ea39e781\") " pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:39.161841 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.161695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-config-volume\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.161841 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.161728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.161841 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:39.161834    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:58:39.161942 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:39.161894    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls podName:e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab nodeName:}" failed. No retries permitted until 2026-09-10 22:58:39.6618796 +0000 UTC m=+36.097934352 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls") pod "dns-default-sqndg" (UID: "e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab") : secret "dns-default-metrics-tls" not found
Sep 10 22:58:39.162038 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.162019    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-tmp-dir\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.162216 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.162199    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-config-volume\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.173508 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.173481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnn7h\" (UniqueName: \"kubernetes.io/projected/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-kube-api-access-dnn7h\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.208482 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.208443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:58:39.208680 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.208443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:58:39.208680 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.208443    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:58:39.211159 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.211137    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 22:58:39.211292 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.211185    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 22:58:39.211292 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.211200    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xmnrw\""
Sep 10 22:58:39.211292 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.211200    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 22:58:39.211292 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.211224    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 22:58:39.211292 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.211231    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hw9xb\""
Sep 10 22:58:39.262677 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.262636    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sxxr\" (UniqueName: \"kubernetes.io/projected/3f5cfece-be37-4c58-80b8-d099ea39e781-kube-api-access-6sxxr\") pod \"ingress-canary-vlhs2\" (UID: \"3f5cfece-be37-4c58-80b8-d099ea39e781\") " pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:39.262842 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.262705    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f5cfece-be37-4c58-80b8-d099ea39e781-cert\") pod \"ingress-canary-vlhs2\" (UID: \"3f5cfece-be37-4c58-80b8-d099ea39e781\") " pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:39.265061 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.265032    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3f5cfece-be37-4c58-80b8-d099ea39e781-cert\") pod \"ingress-canary-vlhs2\" (UID: \"3f5cfece-be37-4c58-80b8-d099ea39e781\") " pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:39.272270 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.272242    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sxxr\" (UniqueName: \"kubernetes.io/projected/3f5cfece-be37-4c58-80b8-d099ea39e781-kube-api-access-6sxxr\") pod \"ingress-canary-vlhs2\" (UID: \"3f5cfece-be37-4c58-80b8-d099ea39e781\") " pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:39.301172 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.301136    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vlhs2"
Sep 10 22:58:39.403198 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.402352    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" event={"ID":"8a03f169-6f4e-4c71-aedd-aa79c1e9e859","Type":"ContainerStarted","Data":"a392d577e86f7894cbec4dc77fa0ba23ce5aeb661e557b5fcdef2ee06a172352"}
Sep 10 22:58:39.427277 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.427183    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-t6gr9" podStartSLOduration=7.837190168 podStartE2EDuration="35.427167703s" totalImagesPullingTime="27.589977535s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:39.426322898 +0000 UTC m=+35.862377671" watchObservedRunningTime="2026-09-10 22:58:39.427167703 +0000 UTC m=+35.863222491"
Sep 10 22:58:39.563888 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.563854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:39.612550 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.612518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vlhs2"]
Sep 10 22:58:39.616145 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:58:39.616117    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5cfece_be37_4c58_80b8_d099ea39e781.slice/crio-983fa8fee3ebba1fd097f8bbca1c2f7265bbaee5da7f845fd968ef54fafb8913 WatchSource:0}: Error finding container 983fa8fee3ebba1fd097f8bbca1c2f7265bbaee5da7f845fd968ef54fafb8913: Status 404 returned error can't find the container with id 983fa8fee3ebba1fd097f8bbca1c2f7265bbaee5da7f845fd968ef54fafb8913
Sep 10 22:58:39.629670 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.629642    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:39.665680 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:39.665649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:39.665866 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:39.665765    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:58:39.665866 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:39.665837    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls podName:e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab nodeName:}" failed. No retries permitted until 2026-09-10 22:58:40.665822751 +0000 UTC m=+37.101877502 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls") pod "dns-default-sqndg" (UID: "e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab") : secret "dns-default-metrics-tls" not found
Sep 10 22:58:40.406462 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:40.406428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlhs2" event={"ID":"3f5cfece-be37-4c58-80b8-d099ea39e781","Type":"ContainerStarted","Data":"983fa8fee3ebba1fd097f8bbca1c2f7265bbaee5da7f845fd968ef54fafb8913"}
Sep 10 22:58:40.673870 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:40.673765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:40.674049 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:40.673919    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:58:40.674049 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:40.673993    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls podName:e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab nodeName:}" failed. No retries permitted until 2026-09-10 22:58:42.673971204 +0000 UTC m=+39.110025976 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls") pod "dns-default-sqndg" (UID: "e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab") : secret "dns-default-metrics-tls" not found
Sep 10 22:58:42.412491 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:42.412287    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vlhs2" event={"ID":"3f5cfece-be37-4c58-80b8-d099ea39e781","Type":"ContainerStarted","Data":"6f56ac178c021aeafd68e674c2a4f16346112ebcc30f9da141c0d3b0bc54c0ae"}
Sep 10 22:58:42.433336 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:42.433232    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vlhs2" podStartSLOduration=1.9711225780000001 podStartE2EDuration="4.433215248s" totalImagesPullingTime="2.46209267s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:58:42.432908293 +0000 UTC m=+38.868963067" watchObservedRunningTime="2026-09-10 22:58:42.433215248 +0000 UTC m=+38.869270041"
Sep 10 22:58:42.689384 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:42.689297    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:42.689529 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:42.689461    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:58:42.689529 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:42.689526    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls podName:e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab nodeName:}" failed. No retries permitted until 2026-09-10 22:58:46.689510513 +0000 UTC m=+43.125565265 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls") pod "dns-default-sqndg" (UID: "e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab") : secret "dns-default-metrics-tls" not found
Sep 10 22:58:43.327325 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:43.327295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:44.784516 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:44.784482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:44.887651 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:44.887618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:46.716922 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:46.716885    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:46.717403 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:46.717021    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:58:46.717403 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:46.717098    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls podName:e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab nodeName:}" failed. No retries permitted until 2026-09-10 22:58:54.717075835 +0000 UTC m=+51.153130592 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls") pod "dns-default-sqndg" (UID: "e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab") : secret "dns-default-metrics-tls" not found
Sep 10 22:58:47.359406 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:47.359373    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:47.444412 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:47.444384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:54.770195 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:54.770138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:58:54.770651 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:54.770278    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:58:54.770651 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:58:54.770368    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls podName:e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab nodeName:}" failed. No retries permitted until 2026-09-10 22:59:10.770350977 +0000 UTC m=+67.206405727 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls") pod "dns-default-sqndg" (UID: "e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab") : secret "dns-default-metrics-tls" not found
Sep 10 22:58:57.667224 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:57.667170    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:58:57.731105 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:58:57.731077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:00.136212 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:00.136179    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:02.384744 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:02.384713    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-cmsj5"
Sep 10 22:59:02.547890 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:02.547853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:02.629450 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:02.629418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:08.728375 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:08.728343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:08.794120 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:08.794091    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:08.865081 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:08.865059    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:09.770566 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.770518    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 22:59:09.780881 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.780859    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 22:59:09.791691 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:09.791668    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 22:59:09.791775 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:09.791736    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs podName:875ba053-4d40-4cb4-b311-39c5ea4a56c8 nodeName:}" failed. No retries permitted until 2026-09-10 23:00:13.791718612 +0000 UTC m=+130.227773363 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs") pod "network-metrics-daemon-fs2ps" (UID: "875ba053-4d40-4cb4-b311-39c5ea4a56c8") : secret "metrics-daemon-secret" not found
Sep 10 22:59:09.871114 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.871080    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:59:09.871211 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.871148    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:59:09.873869 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.873851    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 22:59:09.873950 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.873932    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 22:59:09.884378 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.884343    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 22:59:09.884945 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.884924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/35acf209-c21a-4958-a3e9-0e951197405b-original-pull-secret\") pod \"global-pull-secret-syncer-zd4wd\" (UID: \"35acf209-c21a-4958-a3e9-0e951197405b\") " pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:59:09.895039 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:09.895015    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj599\" (UniqueName: \"kubernetes.io/projected/5cf44708-d6fc-4ec3-9649-400dbed4d913-kube-api-access-sj599\") pod \"network-check-target-dxcff\" (UID: \"5cf44708-d6fc-4ec3-9649-400dbed4d913\") " pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:59:10.129844 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.129750    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xmnrw\""
Sep 10 22:59:10.131524 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.131500    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zd4wd"
Sep 10 22:59:10.138036 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.138014    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:59:10.417753 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.417719    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zd4wd"]
Sep 10 22:59:10.420697 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.420669    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dxcff"]
Sep 10 22:59:10.423238 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:10.423207    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35acf209_c21a_4958_a3e9_0e951197405b.slice/crio-7a34f44f494a70e91209aefdeab3ce61e7180bf01ae7aa9587a0ff4cb5eb6655 WatchSource:0}: Error finding container 7a34f44f494a70e91209aefdeab3ce61e7180bf01ae7aa9587a0ff4cb5eb6655: Status 404 returned error can't find the container with id 7a34f44f494a70e91209aefdeab3ce61e7180bf01ae7aa9587a0ff4cb5eb6655
Sep 10 22:59:10.423714 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:10.423688    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cf44708_d6fc_4ec3_9649_400dbed4d913.slice/crio-cbbb49d486ff60b1901ab67a4021b895719677d3d452f716e48e79f10e1af965 WatchSource:0}: Error finding container cbbb49d486ff60b1901ab67a4021b895719677d3d452f716e48e79f10e1af965: Status 404 returned error can't find the container with id cbbb49d486ff60b1901ab67a4021b895719677d3d452f716e48e79f10e1af965
Sep 10 22:59:10.477252 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.477216    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dxcff" event={"ID":"5cf44708-d6fc-4ec3-9649-400dbed4d913","Type":"ContainerStarted","Data":"cbbb49d486ff60b1901ab67a4021b895719677d3d452f716e48e79f10e1af965"}
Sep 10 22:59:10.478341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.478317    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zd4wd" event={"ID":"35acf209-c21a-4958-a3e9-0e951197405b","Type":"ContainerStarted","Data":"7a34f44f494a70e91209aefdeab3ce61e7180bf01ae7aa9587a0ff4cb5eb6655"}
Sep 10 22:59:10.778277 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:10.778179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:59:10.778652 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:10.778328    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 22:59:10.778652 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:10.778395    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls podName:e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab nodeName:}" failed. No retries permitted until 2026-09-10 22:59:42.778379078 +0000 UTC m=+99.214433833 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls") pod "dns-default-sqndg" (UID: "e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab") : secret "dns-default-metrics-tls" not found
Sep 10 22:59:12.308625 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:12.308585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:12.372720 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:12.372681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:13.404539 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.404504    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"]
Sep 10 22:59:13.406436 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.406410    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:13.408720 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.408694    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Sep 10 22:59:13.408851 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.408693    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Sep 10 22:59:13.409098 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.409078    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:59:13.409299 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.409281    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-kskvf\""
Sep 10 22:59:13.418586 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.418556    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"]
Sep 10 22:59:13.497350 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.497313    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmkn2\" (UniqueName: \"kubernetes.io/projected/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-kube-api-access-jmkn2\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:13.497532 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.497441    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:13.513306 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.513268    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"]
Sep 10 22:59:13.517175 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.515840    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-ff548d955-jd9zz"]
Sep 10 22:59:13.517345 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.517251    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.518242 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.518221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.520504 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520038    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Sep 10 22:59:13.520504 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520051    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Sep 10 22:59:13.520504 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520096    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Sep 10 22:59:13.520504 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520142    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Sep 10 22:59:13.520504 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520187    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-d4hsd\""
Sep 10 22:59:13.520504 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520367    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 22:59:13.520504 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520403    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-vnj2j\""
Sep 10 22:59:13.520931 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520610    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 22:59:13.520931 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.520813    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 22:59:13.526702 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.526683    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 22:59:13.527629 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.527610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"]
Sep 10 22:59:13.530354 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.530330    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ff548d955-jd9zz"]
Sep 10 22:59:13.597874 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.597836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmw2\" (UniqueName: \"kubernetes.io/projected/9676f20c-a73c-4e02-97dd-9c12b6792649-kube-api-access-kzmw2\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.597874 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.597883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-bound-sa-token\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598126 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.597917    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9676f20c-a73c-4e02-97dd-9c12b6792649-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.598126 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598005    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-image-registry-private-configuration\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598126 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598048    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:13.598126 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598075    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598126 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598099    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-certificates\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42bc8a98-b029-4899-8b8f-9219dac75aec-ca-trust-extracted\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:13.598158    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598181    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx5p7\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-kube-api-access-mx5p7\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:13.598230    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls podName:dc670eae-1eaa-4daf-b1d4-12331d9fbb33 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:14.098212987 +0000 UTC m=+70.534267738 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4jfr4" (UID: "dc670eae-1eaa-4daf-b1d4-12331d9fbb33") : secret "samples-operator-tls" not found
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-trusted-ca\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598304    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-installation-pull-secrets\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmkn2\" (UniqueName: \"kubernetes.io/projected/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-kube-api-access-jmkn2\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:13.598444 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.598361    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9676f20c-a73c-4e02-97dd-9c12b6792649-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.611896 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.611863    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmkn2\" (UniqueName: \"kubernetes.io/projected/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-kube-api-access-jmkn2\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:13.699469 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmw2\" (UniqueName: \"kubernetes.io/projected/9676f20c-a73c-4e02-97dd-9c12b6792649-kube-api-access-kzmw2\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.699469 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699439    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-bound-sa-token\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699471    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9676f20c-a73c-4e02-97dd-9c12b6792649-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699499    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-image-registry-private-configuration\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699534    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-certificates\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699600    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42bc8a98-b029-4899-8b8f-9219dac75aec-ca-trust-extracted\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx5p7\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-kube-api-access-mx5p7\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699647    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-trusted-ca\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.699701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699669    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-installation-pull-secrets\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.700281 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.699704    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9676f20c-a73c-4e02-97dd-9c12b6792649-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.700281 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:13.699726    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 22:59:13.700281 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:13.699747    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff548d955-jd9zz: secret "image-registry-tls" not found
Sep 10 22:59:13.700281 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:13.699841    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls podName:42bc8a98-b029-4899-8b8f-9219dac75aec nodeName:}" failed. No retries permitted until 2026-09-10 22:59:14.199818289 +0000 UTC m=+70.635873040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls") pod "image-registry-ff548d955-jd9zz" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec") : secret "image-registry-tls" not found
Sep 10 22:59:13.700492 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.700318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/9676f20c-a73c-4e02-97dd-9c12b6792649-config\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.700571 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.700548    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-certificates\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.700824 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.700793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42bc8a98-b029-4899-8b8f-9219dac75aec-ca-trust-extracted\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.701348 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.701319    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-trusted-ca\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.702488 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.702464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/9676f20c-a73c-4e02-97dd-9c12b6792649-serving-cert\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.702595 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.702577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-image-registry-private-configuration\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.702717 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.702697    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-installation-pull-secrets\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.709044 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.708985    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmw2\" (UniqueName: \"kubernetes.io/projected/9676f20c-a73c-4e02-97dd-9c12b6792649-kube-api-access-kzmw2\") pod \"kube-storage-version-migrator-operator-6c8f84f897-9nj9m\" (UID: \"9676f20c-a73c-4e02-97dd-9c12b6792649\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:13.709526 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.709505    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-bound-sa-token\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.709638 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.709528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx5p7\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-kube-api-access-mx5p7\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:13.768413 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.768376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:13.821875 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.821835    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:13.830571 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:13.830540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"
Sep 10 22:59:14.105572 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:14.104945    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:14.105572 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:14.105183    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:59:14.105572 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:14.105249    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls podName:dc670eae-1eaa-4daf-b1d4-12331d9fbb33 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:15.105229532 +0000 UTC m=+71.541284286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4jfr4" (UID: "dc670eae-1eaa-4daf-b1d4-12331d9fbb33") : secret "samples-operator-tls" not found
Sep 10 22:59:14.206184 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:14.206139    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:14.206419 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:14.206305    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 22:59:14.206419 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:14.206331    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff548d955-jd9zz: secret "image-registry-tls" not found
Sep 10 22:59:14.206419 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:14.206405    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls podName:42bc8a98-b029-4899-8b8f-9219dac75aec nodeName:}" failed. No retries permitted until 2026-09-10 22:59:15.20638275 +0000 UTC m=+71.642437504 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls") pod "image-registry-ff548d955-jd9zz" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec") : secret "image-registry-tls" not found
Sep 10 22:59:15.026430 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.026346    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m"]
Sep 10 22:59:15.029554 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:15.029528    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9676f20c_a73c_4e02_97dd_9c12b6792649.slice/crio-b38ec717988f08efbcfd422ecf86fb4ec2146ac28063dabb826d032152db019a WatchSource:0}: Error finding container b38ec717988f08efbcfd422ecf86fb4ec2146ac28063dabb826d032152db019a: Status 404 returned error can't find the container with id b38ec717988f08efbcfd422ecf86fb4ec2146ac28063dabb826d032152db019a
Sep 10 22:59:15.114246 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.114202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:15.114418 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:15.114356    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:59:15.114459 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:15.114423    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls podName:dc670eae-1eaa-4daf-b1d4-12331d9fbb33 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:17.114407324 +0000 UTC m=+73.550462074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4jfr4" (UID: "dc670eae-1eaa-4daf-b1d4-12331d9fbb33") : secret "samples-operator-tls" not found
Sep 10 22:59:15.214596 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.214551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:15.214819 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:15.214727    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 22:59:15.214819 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:15.214749    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff548d955-jd9zz: secret "image-registry-tls" not found
Sep 10 22:59:15.214899 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:15.214836    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls podName:42bc8a98-b029-4899-8b8f-9219dac75aec nodeName:}" failed. No retries permitted until 2026-09-10 22:59:17.21481913 +0000 UTC m=+73.650873881 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls") pod "image-registry-ff548d955-jd9zz" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec") : secret "image-registry-tls" not found
Sep 10 22:59:15.494646 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.494605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m" event={"ID":"9676f20c-a73c-4e02-97dd-9c12b6792649","Type":"ContainerStarted","Data":"b38ec717988f08efbcfd422ecf86fb4ec2146ac28063dabb826d032152db019a"}
Sep 10 22:59:15.496157 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.496077    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dxcff" event={"ID":"5cf44708-d6fc-4ec3-9649-400dbed4d913","Type":"ContainerStarted","Data":"3ac3a82c369b1750116531d5c77e278c1ac3bc9e9c6bcef42e1d760f9411204b"}
Sep 10 22:59:15.496271 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.496184    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:59:15.497592 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.497569    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zd4wd" event={"ID":"35acf209-c21a-4958-a3e9-0e951197405b","Type":"ContainerStarted","Data":"8f73dbb2204cbdb693fbb0cd4a9da51e6473929d94b799b12ba1f47c9bf433ba"}
Sep 10 22:59:15.514084 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.514028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dxcff" podStartSLOduration=67.175376027 podStartE2EDuration="1m11.514012236s" totalImagesPullingTime="4.338636209s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:15.513527993 +0000 UTC m=+71.949582771" watchObservedRunningTime="2026-09-10 22:59:15.514012236 +0000 UTC m=+71.950067009"
Sep 10 22:59:15.530494 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.530440    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zd4wd" podStartSLOduration=67.184163076 podStartE2EDuration="1m11.530419058s" totalImagesPullingTime="4.346255982s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:15.530318325 +0000 UTC m=+71.966373098" watchObservedRunningTime="2026-09-10 22:59:15.530419058 +0000 UTC m=+71.966473833"
Sep 10 22:59:15.744953 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.744871    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:15.803304 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:15.803266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:17.127929 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.127837    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:17.128280 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:17.127987    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:59:17.128280 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:17.128056    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls podName:dc670eae-1eaa-4daf-b1d4-12331d9fbb33 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:21.128040136 +0000 UTC m=+77.564094892 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4jfr4" (UID: "dc670eae-1eaa-4daf-b1d4-12331d9fbb33") : secret "samples-operator-tls" not found
Sep 10 22:59:17.228726 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.228692    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:17.228903 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:17.228878    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 22:59:17.228946 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:17.228908    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff548d955-jd9zz: secret "image-registry-tls" not found
Sep 10 22:59:17.228979 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:17.228974    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls podName:42bc8a98-b029-4899-8b8f-9219dac75aec nodeName:}" failed. No retries permitted until 2026-09-10 22:59:21.228954364 +0000 UTC m=+77.665009130 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls") pod "image-registry-ff548d955-jd9zz" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec") : secret "image-registry-tls" not found
Sep 10 22:59:17.419550 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.419519    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"]
Sep 10 22:59:17.426315 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.426280    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:17.429365 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.429329    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-pn9mv\""
Sep 10 22:59:17.429517 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.429406    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 22:59:17.429992 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.429973    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 22:59:17.435734 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.435555    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"]
Sep 10 22:59:17.504775 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.504740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m" event={"ID":"9676f20c-a73c-4e02-97dd-9c12b6792649","Type":"ContainerStarted","Data":"caaec742cf61bf961dd560eaca1be1c4e81c6e73fa818919d3d50b6bd0ef154a"}
Sep 10 22:59:17.523279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.523230    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m" podStartSLOduration=2.787167094 podStartE2EDuration="4.523210432s" totalImagesPullingTime="1.736043338s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:17.522335582 +0000 UTC m=+73.958390356" watchObservedRunningTime="2026-09-10 22:59:17.523210432 +0000 UTC m=+73.959265233"
Sep 10 22:59:17.530620 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.530590    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:17.530701 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.530675    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f799298-0931-4049-a366-034434be0df2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:17.544803 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.544752    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:17.631207 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.631163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f799298-0931-4049-a366-034434be0df2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:17.631404 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.631245    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:17.631404 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:17.631371    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 22:59:17.631516 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:17.631447    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert podName:9f799298-0931-4049-a366-034434be0df2 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:18.13142648 +0000 UTC m=+74.567481231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-zrj6s" (UID: "9f799298-0931-4049-a366-034434be0df2") : secret "networking-console-plugin-cert" not found
Sep 10 22:59:17.632029 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.632003    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/9f799298-0931-4049-a366-034434be0df2-nginx-conf\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:17.640809 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.640764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:17.975466 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:17.975436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:18.076684 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:18.076649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:18.135672 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:18.135639    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:18.136133 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:18.135833    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 22:59:18.136133 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:18.135927    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert podName:9f799298-0931-4049-a366-034434be0df2 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:19.135902799 +0000 UTC m=+75.571957554 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-zrj6s" (UID: "9f799298-0931-4049-a366-034434be0df2") : secret "networking-console-plugin-cert" not found
Sep 10 22:59:18.275670 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:18.275596    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:18.370694 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:18.370663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:18.496243 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:18.496213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:18.586379 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:18.586299    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:18.650025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:18.649992    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:19.142766 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.142715    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:19.143183 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:19.142857    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 22:59:19.143183 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:19.142912    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert podName:9f799298-0931-4049-a366-034434be0df2 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:21.142897513 +0000 UTC m=+77.578952264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-zrj6s" (UID: "9f799298-0931-4049-a366-034434be0df2") : secret "networking-console-plugin-cert" not found
Sep 10 22:59:19.854622 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.854587    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rtmpj"]
Sep 10 22:59:19.857458 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.857441    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:19.859466 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.859439    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 22:59:19.860336 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.860318    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 22:59:19.860432 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.860330    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-94bfp\""
Sep 10 22:59:19.860432 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.860328    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 22:59:19.860432 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.860322    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 22:59:19.867630 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.867607    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rtmpj"]
Sep 10 22:59:19.905327 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.905300    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:19.948648 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.948620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c47ebbe0-e696-4383-9060-b77206c6c976-signing-key\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:19.948755 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.948655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zw95\" (UniqueName: \"kubernetes.io/projected/c47ebbe0-e696-4383-9060-b77206c6c976-kube-api-access-2zw95\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:19.948755 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.948685    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c47ebbe0-e696-4383-9060-b77206c6c976-signing-cabundle\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:19.965034 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:19.965010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:20.049411 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.049378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zw95\" (UniqueName: \"kubernetes.io/projected/c47ebbe0-e696-4383-9060-b77206c6c976-kube-api-access-2zw95\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:20.049510 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.049421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c47ebbe0-e696-4383-9060-b77206c6c976-signing-cabundle\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:20.049555 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.049512    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c47ebbe0-e696-4383-9060-b77206c6c976-signing-key\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:20.050158 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.050134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/c47ebbe0-e696-4383-9060-b77206c6c976-signing-cabundle\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:20.051883 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.051865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/c47ebbe0-e696-4383-9060-b77206c6c976-signing-key\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:20.057875 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.057853    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zw95\" (UniqueName: \"kubernetes.io/projected/c47ebbe0-e696-4383-9060-b77206c6c976-kube-api-access-2zw95\") pod \"service-ca-568869fdc5-rtmpj\" (UID: \"c47ebbe0-e696-4383-9060-b77206c6c976\") " pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:20.166442 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.166408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-568869fdc5-rtmpj"
Sep 10 22:59:20.235797 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.235749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:20.245286 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.245240    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hxjdz"]
Sep 10 22:59:20.249552 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.249533    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.251883 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.251773    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 22:59:20.251883 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.251809    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 22:59:20.251883 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.251792    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 22:59:20.251883 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.251774    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 22:59:20.251883 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.251852    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-9dvmk\""
Sep 10 22:59:20.262109 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.262090    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxjdz"]
Sep 10 22:59:20.352069 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.352027    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d1191d1-a838-42a3-a01b-045e73652543-data-volume\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.352236 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.352080    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.352236 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.352131    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5hhb\" (UniqueName: \"kubernetes.io/projected/0d1191d1-a838-42a3-a01b-045e73652543-kube-api-access-k5hhb\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.352236 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.352167    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d1191d1-a838-42a3-a01b-045e73652543-crio-socket\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.352351 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.352234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.355429 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.355408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:20.435229 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.435158    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-568869fdc5-rtmpj"]
Sep 10 22:59:20.438845 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:20.438817    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc47ebbe0_e696_4383_9060_b77206c6c976.slice/crio-c10a9904250e6452edc092b749fff366fc265f16ddf2f3b3f20bc69a2e5dad41 WatchSource:0}: Error finding container c10a9904250e6452edc092b749fff366fc265f16ddf2f3b3f20bc69a2e5dad41: Status 404 returned error can't find the container with id c10a9904250e6452edc092b749fff366fc265f16ddf2f3b3f20bc69a2e5dad41
Sep 10 22:59:20.452855 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.452826    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d1191d1-a838-42a3-a01b-045e73652543-crio-socket\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.452954 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.452889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.452997 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.452953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/0d1191d1-a838-42a3-a01b-045e73652543-crio-socket\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.452997 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.452968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d1191d1-a838-42a3-a01b-045e73652543-data-volume\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.453078 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.453009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.453078 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.453046    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:20.453078 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.453060    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5hhb\" (UniqueName: \"kubernetes.io/projected/0d1191d1-a838-42a3-a01b-045e73652543-kube-api-access-k5hhb\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.453191 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.453107    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:20.953087945 +0000 UTC m=+77.389142702 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:20.453191 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.453157    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:59:20.453272 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.453213    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:20.953200025 +0000 UTC m=+77.389254776 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : configmap "kube-rbac-proxy" not found
Sep 10 22:59:20.453314 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.453269    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/0d1191d1-a838-42a3-a01b-045e73652543-data-volume\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.466427 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.466398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5hhb\" (UniqueName: \"kubernetes.io/projected/0d1191d1-a838-42a3-a01b-045e73652543-kube-api-access-k5hhb\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.517108 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.517075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rtmpj" event={"ID":"c47ebbe0-e696-4383-9060-b77206c6c976","Type":"ContainerStarted","Data":"c10a9904250e6452edc092b749fff366fc265f16ddf2f3b3f20bc69a2e5dad41"}
Sep 10 22:59:20.957524 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.957490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.957699 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:20.957584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:20.957699 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.957659    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:20.957699 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.957676    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:59:20.957898 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.957737    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:21.957721972 +0000 UTC m=+78.393776722 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : configmap "kube-rbac-proxy" not found
Sep 10 22:59:20.957898 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:20.957757    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:21.957747195 +0000 UTC m=+78.393801946 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:21.083415 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:21.083362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:21.159604 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:21.159565    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:21.159820 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:21.159630    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:21.159820 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.159722    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 22:59:21.159820 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.159722    2577 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Sep 10 22:59:21.159820 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.159799    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert podName:9f799298-0931-4049-a366-034434be0df2 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:25.159771314 +0000 UTC m=+81.595826066 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-zrj6s" (UID: "9f799298-0931-4049-a366-034434be0df2") : secret "networking-console-plugin-cert" not found
Sep 10 22:59:21.160019 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.159838    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls podName:dc670eae-1eaa-4daf-b1d4-12331d9fbb33 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:29.159821885 +0000 UTC m=+85.595876649 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls") pod "cluster-samples-operator-86499c89c4-4jfr4" (UID: "dc670eae-1eaa-4daf-b1d4-12331d9fbb33") : secret "samples-operator-tls" not found
Sep 10 22:59:21.260831 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:21.260733    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:21.261216 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.260886    2577 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 22:59:21.261216 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.260904    2577 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-ff548d955-jd9zz: secret "image-registry-tls" not found
Sep 10 22:59:21.261216 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.260966    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls podName:42bc8a98-b029-4899-8b8f-9219dac75aec nodeName:}" failed. No retries permitted until 2026-09-10 22:59:29.260949106 +0000 UTC m=+85.697003857 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls") pod "image-registry-ff548d955-jd9zz" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec") : secret "image-registry-tls" not found
Sep 10 22:59:21.967146 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:21.967103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:21.967343 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:21.967228    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:21.967343 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.967242    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:21.967343 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.967338    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:23.967317767 +0000 UTC m=+80.403372520 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:21.967509 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.967337    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:59:21.967509 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:21.967381    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:23.967370719 +0000 UTC m=+80.403425479 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : configmap "kube-rbac-proxy" not found
Sep 10 22:59:22.524370 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:22.524327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-568869fdc5-rtmpj" event={"ID":"c47ebbe0-e696-4383-9060-b77206c6c976","Type":"ContainerStarted","Data":"22716e279749506b4d77bbecbb425fdad847dd1721498afa41e9c18c975433b1"}
Sep 10 22:59:22.542562 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:22.542505    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-568869fdc5-rtmpj" podStartSLOduration=1.910012858 podStartE2EDuration="3.542485049s" totalImagesPullingTime="1.632472191s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:59:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:22.541072713 +0000 UTC m=+78.977127476" watchObservedRunningTime="2026-09-10 22:59:22.542485049 +0000 UTC m=+78.978539821"
Sep 10 22:59:23.152189 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.152160    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jh4r7_5cebf87f-515b-4607-8e4e-1d53ed85cfda/dns-node-resolver/0.log"
Sep 10 22:59:23.465225 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.465131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:23.549571 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.549533    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:23.649684 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.649650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:23.750717 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.750624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:23.977271 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.977239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:23.986474 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.986445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:23.986631 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:23.986536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:23.986631 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:23.986563    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:59:23.986631 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:23.986623    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:27.986605605 +0000 UTC m=+84.422660362 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : configmap "kube-rbac-proxy" not found
Sep 10 22:59:23.986789 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:23.986648    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:23.986789 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:23.986697    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:27.986680162 +0000 UTC m=+84.422734917 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:24.080697 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:24.080581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:24.153446 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:24.153407    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hrxg_ddf1e8ab-c897-4333-8140-78d9d49493bc/node-ca/0.log"
Sep 10 22:59:24.181624 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:24.181580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:24.278633 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:24.278575    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:24.731541 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:24.731509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:24.952485 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:24.952404    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlhs2_3f5cfece-be37-4c58-80b8-d099ea39e781/serve-healthcheck-canary/0.log"
Sep 10 22:59:25.202148 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:25.202103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:25.202324 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:25.202285    2577 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 22:59:25.202368 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:25.202360    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert podName:9f799298-0931-4049-a366-034434be0df2 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:33.202342399 +0000 UTC m=+89.638397151 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert") pod "networking-console-plugin-5dcbc7df6b-zrj6s" (UID: "9f799298-0931-4049-a366-034434be0df2") : secret "networking-console-plugin-cert" not found
Sep 10 22:59:25.326507 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:25.326469    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:25.555927 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:25.555847    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9nj9m_9676f20c-a73c-4e02-97dd-9c12b6792649/kube-storage-version-migrator-operator/0.log"
Sep 10 22:59:25.728545 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:25.728507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:25.931599 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:25.931564    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:26.529941 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:26.529907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:26.916974 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:26.916946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:27.135305 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:27.135263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:27.190024 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:27.189948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:27.334303 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:27.334253    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:27.434607 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:27.434573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:27.745121 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:27.745086    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:27.804341 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:27.804296    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:27.928178 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:27.928149    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:28.028732 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:28.028651    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:28.028857 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:28.028753    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:28.028857 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:28.028803    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 22:59:28.028934 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:28.028869    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:36.02885328 +0000 UTC m=+92.464908041 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : configmap "kube-rbac-proxy" not found
Sep 10 22:59:28.028979 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:28.028932    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:28.029032 ip-10-0-138-150 kubenswrapper[2577]: E0910 22:59:28.029020    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls podName:0d1191d1-a838-42a3-a01b-045e73652543 nodeName:}" failed. No retries permitted until 2026-09-10 22:59:36.028999984 +0000 UTC m=+92.465054739 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hxjdz" (UID: "0d1191d1-a838-42a3-a01b-045e73652543") : secret "insights-runtime-extractor-tls" not found
Sep 10 22:59:28.362508 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:28.362414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:29.165702 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.165670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:29.237579 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.237524    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:29.240073 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.240046    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/dc670eae-1eaa-4daf-b1d4-12331d9fbb33-samples-operator-tls\") pod \"cluster-samples-operator-86499c89c4-4jfr4\" (UID: \"dc670eae-1eaa-4daf-b1d4-12331d9fbb33\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:29.317229 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.317188    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"
Sep 10 22:59:29.338928 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.338897    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:29.341355 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.341334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"image-registry-ff548d955-jd9zz\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") " pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:29.368134 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.367488    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:29.437956 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.437845    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:29.566794 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.566751    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:29.610389 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.610355    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4"]
Sep 10 22:59:29.672423 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.672395    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:29.712258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.712178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-ff548d955-jd9zz"]
Sep 10 22:59:29.715173 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:29.715141    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42bc8a98_b029_4899_8b8f_9219dac75aec.slice/crio-0e4635728d1fa278e752faf0ef87d3c2f59b0d91a5aaa8fcb74b37eae36a9fcf WatchSource:0}: Error finding container 0e4635728d1fa278e752faf0ef87d3c2f59b0d91a5aaa8fcb74b37eae36a9fcf: Status 404 returned error can't find the container with id 0e4635728d1fa278e752faf0ef87d3c2f59b0d91a5aaa8fcb74b37eae36a9fcf
Sep 10 22:59:29.932687 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:29.932657    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:30.332494 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:30.332454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:30.547308 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:30.547263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4" event={"ID":"dc670eae-1eaa-4daf-b1d4-12331d9fbb33","Type":"ContainerStarted","Data":"6789135f94eadb5990ff372c3738d110842459db5aabd1739a6d754ad9340cac"}
Sep 10 22:59:30.549453 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:30.549421    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff548d955-jd9zz" event={"ID":"42bc8a98-b029-4899-8b8f-9219dac75aec","Type":"ContainerStarted","Data":"b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196"}
Sep 10 22:59:30.549586 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:30.549473    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff548d955-jd9zz" event={"ID":"42bc8a98-b029-4899-8b8f-9219dac75aec","Type":"ContainerStarted","Data":"0e4635728d1fa278e752faf0ef87d3c2f59b0d91a5aaa8fcb74b37eae36a9fcf"}
Sep 10 22:59:30.549649 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:30.549588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:30.573117 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:30.573056    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-ff548d955-jd9zz" podStartSLOduration=17.573040809 podStartE2EDuration="17.573040809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:59:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:30.572696249 +0000 UTC m=+87.008751059" watchObservedRunningTime="2026-09-10 22:59:30.573040809 +0000 UTC m=+87.009095581"
Sep 10 22:59:31.134262 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:31.134214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:31.196711 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:31.196674    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:31.328693 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:31.328647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:31.531291 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:31.531261    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:31.592689 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:31.592652    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:31.739233 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:31.739189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:31.941342 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:31.941287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:32.328808 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:32.328715    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:32.558036 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:32.557999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4" event={"ID":"dc670eae-1eaa-4daf-b1d4-12331d9fbb33","Type":"ContainerStarted","Data":"ace3b790e3f268bad566aa12652f256fbd96e1e42605a072bc96667f61ed58f6"}
Sep 10 22:59:32.558206 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:32.558065    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4" event={"ID":"dc670eae-1eaa-4daf-b1d4-12331d9fbb33","Type":"ContainerStarted","Data":"a7faa61a9ee0e4624b9aba1a18a0da536650c76dd4c94451954d9b6d68276336"}
Sep 10 22:59:32.581771 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:32.581659    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-86499c89c4-4jfr4" podStartSLOduration=17.644701439 podStartE2EDuration="19.581640214s" totalImagesPullingTime="1.936938775s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:59:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:32.580909218 +0000 UTC m=+89.016963993" watchObservedRunningTime="2026-09-10 22:59:32.581640214 +0000 UTC m=+89.017695034"
Sep 10 22:59:33.142011 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.141980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:33.272025 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.271988    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:33.274356 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.274326    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9f799298-0931-4049-a366-034434be0df2-networking-console-plugin-cert\") pod \"networking-console-plugin-5dcbc7df6b-zrj6s\" (UID: \"9f799298-0931-4049-a366-034434be0df2\") " pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:33.332686 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.332650    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:33.336343 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.336324    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"
Sep 10 22:59:33.608547 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.608496    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s"]
Sep 10 22:59:33.611569 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:33.611539    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f799298_0931_4049_a366_034434be0df2.slice/crio-d1bd130dc2c5fda4e1f0614848f1f78d9921418193f06b2dee97826cab1a1ab3 WatchSource:0}: Error finding container d1bd130dc2c5fda4e1f0614848f1f78d9921418193f06b2dee97826cab1a1ab3: Status 404 returned error can't find the container with id d1bd130dc2c5fda4e1f0614848f1f78d9921418193f06b2dee97826cab1a1ab3
Sep 10 22:59:33.744837 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.744805    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:33.938448 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:33.938415    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:34.135560 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:34.135509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:34.565094 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:34.565057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s" event={"ID":"9f799298-0931-4049-a366-034434be0df2","Type":"ContainerStarted","Data":"d1bd130dc2c5fda4e1f0614848f1f78d9921418193f06b2dee97826cab1a1ab3"}
Sep 10 22:59:35.569486 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:35.569449    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s" event={"ID":"9f799298-0931-4049-a366-034434be0df2","Type":"ContainerStarted","Data":"bedb4cb0bed3b398976f241c86d395874ac8ee31638c2863d4b4d9478b75e842"}
Sep 10 22:59:35.596313 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:35.596265    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-5dcbc7df6b-zrj6s" podStartSLOduration=17.561468763 podStartE2EDuration="18.596248042s" totalImagesPullingTime="1.034779279s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:59:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:35.594690515 +0000 UTC m=+92.030745287" watchObservedRunningTime="2026-09-10 22:59:35.596248042 +0000 UTC m=+92.032302836"
Sep 10 22:59:36.094350 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.094309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:36.094550 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.094405    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:36.095011 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.094987    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/0d1191d1-a838-42a3-a01b-045e73652543-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:36.096590 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.096567    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/0d1191d1-a838-42a3-a01b-045e73652543-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hxjdz\" (UID: \"0d1191d1-a838-42a3-a01b-045e73652543\") " pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:36.158729 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.158695    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hxjdz"
Sep 10 22:59:36.434298 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.434270    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hxjdz"]
Sep 10 22:59:36.436729 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:36.436703    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0d1191d1_a838_42a3_a01b_045e73652543.slice/crio-cae62c502f757ab92354e601c14c5127c039b4f06f688ee23c4b7d687e5049b4 WatchSource:0}: Error finding container cae62c502f757ab92354e601c14c5127c039b4f06f688ee23c4b7d687e5049b4: Status 404 returned error can't find the container with id cae62c502f757ab92354e601c14c5127c039b4f06f688ee23c4b7d687e5049b4
Sep 10 22:59:36.574265 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.574232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxjdz" event={"ID":"0d1191d1-a838-42a3-a01b-045e73652543","Type":"ContainerStarted","Data":"6b733d6763a4f8d4c3a5c635ca2d488cad9e013b0da8bfbaa466285a49b1f865"}
Sep 10 22:59:36.574655 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:36.574290    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxjdz" event={"ID":"0d1191d1-a838-42a3-a01b-045e73652543","Type":"ContainerStarted","Data":"cae62c502f757ab92354e601c14c5127c039b4f06f688ee23c4b7d687e5049b4"}
Sep 10 22:59:37.580040 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:37.579956    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxjdz" event={"ID":"0d1191d1-a838-42a3-a01b-045e73652543","Type":"ContainerStarted","Data":"fd5ea2d84345a7849c1d1d27a8ca605ca4405eb76c620849c7ac57bfcca39821"}
Sep 10 22:59:38.605896 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.605863    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"]
Sep 10 22:59:38.608728 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.608711    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"]
Sep 10 22:59:38.608878 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.608859    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.612034 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.612012    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.618258 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.618234    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 22:59:38.618443 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.618430    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 22:59:38.618501 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.618454    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 22:59:38.618982 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.618965    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 22:59:38.621192 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.621175    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 22:59:38.626071 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.626054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 22:59:38.626179 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.626086    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-bcn42\""
Sep 10 22:59:38.626179 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.626055    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 22:59:38.626290 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.626222    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 22:59:38.646378 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.646350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"]
Sep 10 22:59:38.655301 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.655276    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff548d955-jd9zz"]
Sep 10 22:59:38.672047 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.672021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"]
Sep 10 22:59:38.709740 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.709708    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:38.715867 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.715843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.715973 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.715876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/327efe60-0bf5-4d96-9945-02a1cf0f246c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-579684d77b-9bvmg\" (UID: \"327efe60-0bf5-4d96-9945-02a1cf0f246c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.715973 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.715951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-hub\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.716055 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.716006    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e89a3-6d22-4b08-8b07-f7305c184c58-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.716055 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.716028    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-ca\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.716125 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.716110    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.716160 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.716138    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8qkb\" (UniqueName: \"kubernetes.io/projected/327efe60-0bf5-4d96-9945-02a1cf0f246c-kube-api-access-b8qkb\") pod \"managed-serviceaccount-addon-agent-579684d77b-9bvmg\" (UID: \"327efe60-0bf5-4d96-9945-02a1cf0f246c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.716194 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.716156    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9t2x\" (UniqueName: \"kubernetes.io/projected/850e89a3-6d22-4b08-8b07-f7305c184c58-kube-api-access-j9t2x\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.793591 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.793510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:38.817114 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e89a3-6d22-4b08-8b07-f7305c184c58-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.817114 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-ca\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.817331 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.817331 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817178    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8qkb\" (UniqueName: \"kubernetes.io/projected/327efe60-0bf5-4d96-9945-02a1cf0f246c-kube-api-access-b8qkb\") pod \"managed-serviceaccount-addon-agent-579684d77b-9bvmg\" (UID: \"327efe60-0bf5-4d96-9945-02a1cf0f246c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.817331 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9t2x\" (UniqueName: \"kubernetes.io/projected/850e89a3-6d22-4b08-8b07-f7305c184c58-kube-api-access-j9t2x\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.817331 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.817331 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817247    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/327efe60-0bf5-4d96-9945-02a1cf0f246c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-579684d77b-9bvmg\" (UID: \"327efe60-0bf5-4d96-9945-02a1cf0f246c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.817331 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.817293    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-hub\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.818091 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.818062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/850e89a3-6d22-4b08-8b07-f7305c184c58-additional-service-ca\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.819841 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.819809    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.819940 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.819886    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-ca\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.820037 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.820016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.820172 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.820157    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/850e89a3-6d22-4b08-8b07-f7305c184c58-hub\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.820382 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.820362    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/327efe60-0bf5-4d96-9945-02a1cf0f246c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-579684d77b-9bvmg\" (UID: \"327efe60-0bf5-4d96-9945-02a1cf0f246c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.831156 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.831131    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9t2x\" (UniqueName: \"kubernetes.io/projected/850e89a3-6d22-4b08-8b07-f7305c184c58-kube-api-access-j9t2x\") pod \"cluster-proxy-proxy-agent-79fdf55649-khncp\" (UID: \"850e89a3-6d22-4b08-8b07-f7305c184c58\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:38.832255 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.832234    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8qkb\" (UniqueName: \"kubernetes.io/projected/327efe60-0bf5-4d96-9945-02a1cf0f246c-kube-api-access-b8qkb\") pod \"managed-serviceaccount-addon-agent-579684d77b-9bvmg\" (UID: \"327efe60-0bf5-4d96-9945-02a1cf0f246c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.899636 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.899602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:38.932495 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.932472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"
Sep 10 22:59:38.937516 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:38.937498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"
Sep 10 22:59:39.203766 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:39.203728    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg"]
Sep 10 22:59:39.206418 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:39.206394    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod327efe60_0bf5_4d96_9945_02a1cf0f246c.slice/crio-4fe2e9f0920e4522959adfa7595b6706dcee243b4bf97df9e08fdf95436d576b WatchSource:0}: Error finding container 4fe2e9f0920e4522959adfa7595b6706dcee243b4bf97df9e08fdf95436d576b: Status 404 returned error can't find the container with id 4fe2e9f0920e4522959adfa7595b6706dcee243b4bf97df9e08fdf95436d576b
Sep 10 22:59:39.210754 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:39.210733    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp"]
Sep 10 22:59:39.213573 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:39.213552    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod850e89a3_6d22_4b08_8b07_f7305c184c58.slice/crio-47904e409cf9d8393fe605211cd480b9f27625a80a6ebf2c1f832b8c18d786c6 WatchSource:0}: Error finding container 47904e409cf9d8393fe605211cd480b9f27625a80a6ebf2c1f832b8c18d786c6: Status 404 returned error can't find the container with id 47904e409cf9d8393fe605211cd480b9f27625a80a6ebf2c1f832b8c18d786c6
Sep 10 22:59:39.588916 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:39.588832    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hxjdz" event={"ID":"0d1191d1-a838-42a3-a01b-045e73652543","Type":"ContainerStarted","Data":"b45dd9e495d37f95b94f063a7f5f92f03596c3f97f77c4d4e031d0c55f8c4a6e"}
Sep 10 22:59:39.590159 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:39.590133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg" event={"ID":"327efe60-0bf5-4d96-9945-02a1cf0f246c","Type":"ContainerStarted","Data":"4fe2e9f0920e4522959adfa7595b6706dcee243b4bf97df9e08fdf95436d576b"}
Sep 10 22:59:39.591108 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:39.591087    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp" event={"ID":"850e89a3-6d22-4b08-8b07-f7305c184c58","Type":"ContainerStarted","Data":"47904e409cf9d8393fe605211cd480b9f27625a80a6ebf2c1f832b8c18d786c6"}
Sep 10 22:59:39.608090 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:39.608033    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hxjdz" podStartSLOduration=17.620292621 podStartE2EDuration="19.608019333s" totalImagesPullingTime="1.987726712s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 22:59:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:39.607230113 +0000 UTC m=+96.043284886" watchObservedRunningTime="2026-09-10 22:59:39.608019333 +0000 UTC m=+96.044074105"
Sep 10 22:59:42.854618 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:42.854577    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:59:42.857224 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:42.857195    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab-metrics-tls\") pod \"dns-default-sqndg\" (UID: \"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab\") " pod="openshift-dns/dns-default-sqndg"
Sep 10 22:59:42.880739 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:42.880711    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-677w4\""
Sep 10 22:59:42.888672 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:42.888650    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sqndg"
Sep 10 22:59:43.376719 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.376626    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sqndg"]
Sep 10 22:59:43.381404 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:43.381378    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3b6ccc7_12b0_4b07_8ea8_32e4295e19ab.slice/crio-30b19b3a7a9c111bb504b8f98eac33595b64319007368ad5824e8b9767d076f0 WatchSource:0}: Error finding container 30b19b3a7a9c111bb504b8f98eac33595b64319007368ad5824e8b9767d076f0: Status 404 returned error can't find the container with id 30b19b3a7a9c111bb504b8f98eac33595b64319007368ad5824e8b9767d076f0
Sep 10 22:59:43.610771 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.610740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp" event={"ID":"850e89a3-6d22-4b08-8b07-f7305c184c58","Type":"ContainerStarted","Data":"81d9ee0090754776d7774647ec2e71ec026ebd4d5e3271250f3b188156346811"}
Sep 10 22:59:43.610771 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.610795    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp" event={"ID":"850e89a3-6d22-4b08-8b07-f7305c184c58","Type":"ContainerStarted","Data":"6c9ed130ede1a2157747f0743772c70fee1669c05999f93eda7a10fe4e2182db"}
Sep 10 22:59:43.611031 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.610814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp" event={"ID":"850e89a3-6d22-4b08-8b07-f7305c184c58","Type":"ContainerStarted","Data":"8496bf4b7d16f4e51ae368386ebea6827f523a8aa45d44b03f61d65014ac4f68"}
Sep 10 22:59:43.611905 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.611881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqndg" event={"ID":"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab","Type":"ContainerStarted","Data":"30b19b3a7a9c111bb504b8f98eac33595b64319007368ad5824e8b9767d076f0"}
Sep 10 22:59:43.613107 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.613088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg" event={"ID":"327efe60-0bf5-4d96-9945-02a1cf0f246c","Type":"ContainerStarted","Data":"89ad70b0a2ac6e755ed6bfd60b45e0f7171b652d0fe2bbf20d3a90a471108873"}
Sep 10 22:59:43.629350 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.629254    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-79fdf55649-khncp" podStartSLOduration=1.725272734 podStartE2EDuration="5.629237704s" totalImagesPullingTime="3.90396497s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:43.628121786 +0000 UTC m=+100.064176559" watchObservedRunningTime="2026-09-10 22:59:43.629237704 +0000 UTC m=+100.065292477"
Sep 10 22:59:43.642943 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:43.642892    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-579684d77b-9bvmg" podStartSLOduration=1.745734021 podStartE2EDuration="5.642877564s" totalImagesPullingTime="3.897143543s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:59:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:43.642056755 +0000 UTC m=+100.078111527" watchObservedRunningTime="2026-09-10 22:59:43.642877564 +0000 UTC m=+100.078932335"
Sep 10 22:59:45.621902 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:45.621866    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqndg" event={"ID":"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab","Type":"ContainerStarted","Data":"b1b745abad15e98e6476172dbf054cb5577ff7474b6d69ded628c5e0100588bf"}
Sep 10 22:59:45.622279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:45.621924    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sqndg" event={"ID":"e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab","Type":"ContainerStarted","Data":"866f2511bcd744d0c53438a7514eb9e34cb0fcc56270b41b8d5fea8938365e23"}
Sep 10 22:59:45.622279 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:45.621962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sqndg"
Sep 10 22:59:45.639850 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:45.639800    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sqndg" podStartSLOduration=65.988654717 podStartE2EDuration="1m7.63976888s" totalImagesPullingTime="1.651114163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:58:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:45.638960973 +0000 UTC m=+102.075015744" watchObservedRunningTime="2026-09-10 22:59:45.63976888 +0000 UTC m=+102.075823650"
Sep 10 22:59:46.503067 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:46.503037    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dxcff"
Sep 10 22:59:47.708648 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:47.708615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:47.771506 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:47.771473    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:47.881467 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:47.881438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:48.326431 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:48.326397    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:48.661498 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:48.661458    2577 patch_prober.go:28] interesting pod/image-registry-ff548d955-jd9zz container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 22:59:48.661658 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:48.661516    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-ff548d955-jd9zz" podUID="42bc8a98-b029-4899-8b8f-9219dac75aec" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 22:59:50.930513 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:50.930479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:50.999919 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:50.999880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:51.099895 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:51.099862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:53.574066 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:53.574031    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.012865 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.012824    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.111794 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.111672    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.217775 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.217719    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.331498 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.331376    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.398575 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.398541    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-cmmlc"]
Sep 10 22:59:55.402944 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.402919    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.405473 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.405444    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 22:59:55.405865 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.405845    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 22:59:55.405976 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.405870    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 22:59:55.405976 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.405913    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 22:59:55.405976 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.405949    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 22:59:55.406186 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.406172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-22fhz\""
Sep 10 22:59:55.406383 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.406369    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 22:59:55.428991 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.428901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.448475 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448430    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-root\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448475 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448485    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448767 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-textfile\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448767 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-tls\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448767 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448594    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-sys\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448767 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448616    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-wtmp\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448767 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9plkv\" (UniqueName: \"kubernetes.io/projected/f9b8494b-9f4b-41fd-b201-dd6aed67f685-kube-api-access-9plkv\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448767 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448668    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.448767 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.448711    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b8494b-9f4b-41fd-b201-dd6aed67f685-metrics-client-ca\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.500921 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.500887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.549594 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.549798 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549608    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b8494b-9f4b-41fd-b201-dd6aed67f685-metrics-client-ca\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.549798 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-root\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.549798 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549695    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.549798 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549754    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-textfile\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550018 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549774    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-root\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550018 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-tls\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550018 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549875    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-sys\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550018 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549903    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-wtmp\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550018 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549930    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9plkv\" (UniqueName: \"kubernetes.io/projected/f9b8494b-9f4b-41fd-b201-dd6aed67f685-kube-api-access-9plkv\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550018 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.549996    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-sys\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550304 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.550137    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-wtmp\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550304 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.550205    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-textfile\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550434 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.550409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-accelerators-collector-config\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.550916 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.550891    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f9b8494b-9f4b-41fd-b201-dd6aed67f685-metrics-client-ca\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.552064 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.552044    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-tls\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.552278 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.552259    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f9b8494b-9f4b-41fd-b201-dd6aed67f685-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.557128 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.557099    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.559124 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.559103    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9plkv\" (UniqueName: \"kubernetes.io/projected/f9b8494b-9f4b-41fd-b201-dd6aed67f685-kube-api-access-9plkv\") pod \"node-exporter-cmmlc\" (UID: \"f9b8494b-9f4b-41fd-b201-dd6aed67f685\") " pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.627926 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.627851    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sqndg"
Sep 10 22:59:55.713924 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.713894    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-cmmlc"
Sep 10 22:59:55.728160 ip-10-0-138-150 kubenswrapper[2577]: W0910 22:59:55.728118    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9b8494b_9f4b_41fd_b201_dd6aed67f685.slice/crio-0ff9d85ab350453a554e735d33898bdb6923110758b6dfe816e9a110ab6cff5a WatchSource:0}: Error finding container 0ff9d85ab350453a554e735d33898bdb6923110758b6dfe816e9a110ab6cff5a: Status 404 returned error can't find the container with id 0ff9d85ab350453a554e735d33898bdb6923110758b6dfe816e9a110ab6cff5a
Sep 10 22:59:55.742250 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.742229    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.840836 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.840802    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:55.965789 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:55.965728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:56.349272 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:56.349190    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:56.456453 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:56.456416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:56.662933 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:56.662901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmmlc" event={"ID":"f9b8494b-9f4b-41fd-b201-dd6aed67f685","Type":"ContainerStarted","Data":"0ff9d85ab350453a554e735d33898bdb6923110758b6dfe816e9a110ab6cff5a"}
Sep 10 22:59:56.672056 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:56.672021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:57.668378 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:57.668341    2577 generic.go:356] "Generic (PLEG): container finished" podID="f9b8494b-9f4b-41fd-b201-dd6aed67f685" containerID="3d7129d23535c71a3608cd2353758b5398bbd430eff92d0fcdf20196ba728cc7" exitCode=0
Sep 10 22:59:57.668902 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:57.668389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmmlc" event={"ID":"f9b8494b-9f4b-41fd-b201-dd6aed67f685","Type":"ContainerDied","Data":"3d7129d23535c71a3608cd2353758b5398bbd430eff92d0fcdf20196ba728cc7"}
Sep 10 22:59:57.899245 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:57.899196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:57.964173 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:57.964079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:58.047636 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.047599    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:58.150326 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.150295    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:58.400443 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.400292    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:58.501257 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.501222    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:58.660684 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.660658    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 22:59:58.674859 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.674829    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmmlc" event={"ID":"f9b8494b-9f4b-41fd-b201-dd6aed67f685","Type":"ContainerStarted","Data":"a27d661bf6acba08696f45ba1af0ce5fd144206c17fad61084ca4b8ec6e3b3c2"}
Sep 10 22:59:58.675218 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.674886    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-cmmlc" event={"ID":"f9b8494b-9f4b-41fd-b201-dd6aed67f685","Type":"ContainerStarted","Data":"09ee74e4c3569326bd9484b9c406b6e39241ae439ea8489cd61c2978eb228315"}
Sep 10 22:59:58.709510 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.709440    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-cmmlc" podStartSLOduration=2.659795911 podStartE2EDuration="3.709418956s" totalImagesPullingTime="987.102124ms" totalInitContainerRuntime="62.520921ms" isStatefulPod=true podCreationTimestamp="2026-09-10 22:59:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 22:59:58.708160153 +0000 UTC m=+115.144214949" watchObservedRunningTime="2026-09-10 22:59:58.709418956 +0000 UTC m=+115.145473729"
Sep 10 22:59:58.859627 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.859595    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 22:59:58.938715 ip-10-0-138-150 kubenswrapper[2577]: I0910 22:59:58.938639    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:00.135129 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:00.135087    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:00.339279 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:00.339245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:00.945280 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:00.945246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:01.045561 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:01.045511    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:01.227064 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:01.226975    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:01.367374 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:01.367342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:01.671170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:01.671134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:01.832381 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:01.832352    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:03.673680 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:03.673636    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-ff548d955-jd9zz" podUID="42bc8a98-b029-4899-8b8f-9219dac75aec" containerName="registry" containerID="cri-o://b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196" gracePeriod=30
Sep 10 23:00:03.914287 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:03.914254    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 23:00:04.012245 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012147    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-trusted-ca\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012245 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012210    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-installation-pull-secrets\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012245 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012234    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012272    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-image-registry-private-configuration\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012306    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-certificates\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012338    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42bc8a98-b029-4899-8b8f-9219dac75aec-ca-trust-extracted\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012353    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mx5p7\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-kube-api-access-mx5p7\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012369    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-bound-sa-token\") pod \"42bc8a98-b029-4899-8b8f-9219dac75aec\" (UID: \"42bc8a98-b029-4899-8b8f-9219dac75aec\") "
Sep 10 23:00:04.012824 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012630    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:00:04.012886 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.012873    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:00:04.014889 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.014862    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:00:04.015028 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.014904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-kube-api-access-mx5p7" (OuterVolumeSpecName: "kube-api-access-mx5p7") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "kube-api-access-mx5p7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:00:04.015092 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.015042    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:00:04.015181 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.015159    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:00:04.015231 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.015170    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:00:04.021248 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.021216    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42bc8a98-b029-4899-8b8f-9219dac75aec-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "42bc8a98-b029-4899-8b8f-9219dac75aec" (UID: "42bc8a98-b029-4899-8b8f-9219dac75aec"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:00:04.113664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113623    2577 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/42bc8a98-b029-4899-8b8f-9219dac75aec-ca-trust-extracted\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.113664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113652    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx5p7\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-kube-api-access-mx5p7\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.113664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113662    2577 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-bound-sa-token\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.113664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113672    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-trusted-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.114033 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113683    2577 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-installation-pull-secrets\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.114033 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113691    2577 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.114033 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113700    2577 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/42bc8a98-b029-4899-8b8f-9219dac75aec-image-registry-private-configuration\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.114033 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.113709    2577 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/42bc8a98-b029-4899-8b8f-9219dac75aec-registry-certificates\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:04.141658 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.141624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:04.249038 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.249002    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:04.697932 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.697895    2577 generic.go:356] "Generic (PLEG): container finished" podID="42bc8a98-b029-4899-8b8f-9219dac75aec" containerID="b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196" exitCode=0
Sep 10 23:00:04.698405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.697964    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-ff548d955-jd9zz"
Sep 10 23:00:04.698405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.697979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff548d955-jd9zz" event={"ID":"42bc8a98-b029-4899-8b8f-9219dac75aec","Type":"ContainerDied","Data":"b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196"}
Sep 10 23:00:04.698405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.698033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-ff548d955-jd9zz" event={"ID":"42bc8a98-b029-4899-8b8f-9219dac75aec","Type":"ContainerDied","Data":"0e4635728d1fa278e752faf0ef87d3c2f59b0d91a5aaa8fcb74b37eae36a9fcf"}
Sep 10 23:00:04.698405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.698055    2577 scope.go:117] "RemoveContainer" containerID="b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196"
Sep 10 23:00:04.706064 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.706041    2577 scope.go:117] "RemoveContainer" containerID="b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196"
Sep 10 23:00:04.706365 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:00:04.706343    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196\": container with ID starting with b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196 not found: ID does not exist" containerID="b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196"
Sep 10 23:00:04.706436 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.706379    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196"} err="failed to get container status \"b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196\": rpc error: code = NotFound desc = could not find container \"b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196\": container with ID starting with b615eec76c67fb2ef7447d4542032316b78a44a32a0a468b8bfed24bb8ace196 not found: ID does not exist"
Sep 10 23:00:04.723495 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.723461    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-ff548d955-jd9zz"]
Sep 10 23:00:04.727334 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:04.727310    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-ff548d955-jd9zz"]
Sep 10 23:00:06.213221 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:06.213185    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42bc8a98-b029-4899-8b8f-9219dac75aec" path="/var/lib/kubelet/pods/42bc8a98-b029-4899-8b8f-9219dac75aec/volumes"
Sep 10 23:00:06.904347 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:06.904263    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:06.970345 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:06.970310    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:07.861475 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:07.861445    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:07.958449 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:07.958414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:08.056042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:08.056009    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:08.348324 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:08.348288    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:08.528501 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:08.528469    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:08.630641 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:08.630551    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:08.958470 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:08.958437    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:09.069654 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:09.069624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:09.137345 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:09.137311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:09.333668 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:09.333589    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:09.541460 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:09.541432    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:09.640051 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:09.639966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:10.065571 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:10.065535    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:10.129570 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:10.129536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:12.332547 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.332510    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58575c88cc-h686p"]
Sep 10 23:00:12.332966 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.332769    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42bc8a98-b029-4899-8b8f-9219dac75aec" containerName="registry"
Sep 10 23:00:12.332966 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.332822    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="42bc8a98-b029-4899-8b8f-9219dac75aec" containerName="registry"
Sep 10 23:00:12.332966 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.332873    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="42bc8a98-b029-4899-8b8f-9219dac75aec" containerName="registry"
Sep 10 23:00:12.402428 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.402394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58575c88cc-h686p"]
Sep 10 23:00:12.402587 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.402490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.405926 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.405887    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-29xcp\""
Sep 10 23:00:12.405926 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.405906    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 23:00:12.405926 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.405927    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 23:00:12.406169 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.405951    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 23:00:12.406169 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.405990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 23:00:12.406348 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.406329    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 23:00:12.406461 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.406349    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 23:00:12.406461 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.406356    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 23:00:12.480417 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.480380    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-oauth-config\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.480417 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.480414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-oauth-serving-cert\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.480676 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.480445    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-serving-cert\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.480676 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.480522    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jmdr\" (UniqueName: \"kubernetes.io/projected/3ae5649b-b75b-47c2-9635-37dae8f603e5-kube-api-access-7jmdr\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.480676 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.480559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-service-ca\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.480676 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.480595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-config\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.581440 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.581393    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jmdr\" (UniqueName: \"kubernetes.io/projected/3ae5649b-b75b-47c2-9635-37dae8f603e5-kube-api-access-7jmdr\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.581440 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.581436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-service-ca\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.581674 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.581469    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-config\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.581674 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.581500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-oauth-config\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.581674 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.581516    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-oauth-serving-cert\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.581674 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.581547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-serving-cert\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.582373 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.582332    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-config\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.582373 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.582352    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-oauth-serving-cert\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.582508 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.582357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-service-ca\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.584053 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.583997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-oauth-config\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.584271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.584250    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-serving-cert\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.590236 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.590211    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jmdr\" (UniqueName: \"kubernetes.io/projected/3ae5649b-b75b-47c2-9635-37dae8f603e5-kube-api-access-7jmdr\") pod \"console-58575c88cc-h686p\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") " pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:12.711266 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:12.711219    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:13.008344 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:13.008312    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58575c88cc-h686p"]
Sep 10 23:00:13.011906 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:00:13.011875    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ae5649b_b75b_47c2_9635_37dae8f603e5.slice/crio-06841dba423a37c7dfa6b70992ce3f801fed82524aa3ac5d200cd1514f5225be WatchSource:0}: Error finding container 06841dba423a37c7dfa6b70992ce3f801fed82524aa3ac5d200cd1514f5225be: Status 404 returned error can't find the container with id 06841dba423a37c7dfa6b70992ce3f801fed82524aa3ac5d200cd1514f5225be
Sep 10 23:00:13.729641 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:13.729607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58575c88cc-h686p" event={"ID":"3ae5649b-b75b-47c2-9635-37dae8f603e5","Type":"ContainerStarted","Data":"06841dba423a37c7dfa6b70992ce3f801fed82524aa3ac5d200cd1514f5225be"}
Sep 10 23:00:13.792798 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:13.792748    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 23:00:13.795444 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:13.795419    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/875ba053-4d40-4cb4-b311-39c5ea4a56c8-metrics-certs\") pod \"network-metrics-daemon-fs2ps\" (UID: \"875ba053-4d40-4cb4-b311-39c5ea4a56c8\") " pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 23:00:14.021189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:14.021104    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hw9xb\""
Sep 10 23:00:14.029825 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:14.029773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-fs2ps"
Sep 10 23:00:14.316692 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:14.316612    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-fs2ps"]
Sep 10 23:00:14.321563 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:00:14.321530    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod875ba053_4d40_4cb4_b311_39c5ea4a56c8.slice/crio-918944c5ede3dc5d6811e27338df2016f77d530b730563da0cbe78231382f886 WatchSource:0}: Error finding container 918944c5ede3dc5d6811e27338df2016f77d530b730563da0cbe78231382f886: Status 404 returned error can't find the container with id 918944c5ede3dc5d6811e27338df2016f77d530b730563da0cbe78231382f886
Sep 10 23:00:14.734893 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:14.734857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs2ps" event={"ID":"875ba053-4d40-4cb4-b311-39c5ea4a56c8","Type":"ContainerStarted","Data":"918944c5ede3dc5d6811e27338df2016f77d530b730563da0cbe78231382f886"}
Sep 10 23:00:15.945645 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:15.945608    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:16.003134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.003104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:16.533479 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.533446    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:16.597119 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.597021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:16.744335 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.744296    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58575c88cc-h686p" event={"ID":"3ae5649b-b75b-47c2-9635-37dae8f603e5","Type":"ContainerStarted","Data":"2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b"}
Sep 10 23:00:16.746064 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.746042    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs2ps" event={"ID":"875ba053-4d40-4cb4-b311-39c5ea4a56c8","Type":"ContainerStarted","Data":"a2504821f7de48adf91f180717e0b8dedf23c46cc2b6ca68f45a409d4abda1da"}
Sep 10 23:00:16.746185 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.746080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-fs2ps" event={"ID":"875ba053-4d40-4cb4-b311-39c5ea4a56c8","Type":"ContainerStarted","Data":"fc228c3258de8ad6e9478f5f785b073f34c6f5f0c9dc060ced88aa2d9b64b10f"}
Sep 10 23:00:16.763832 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.763756    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58575c88cc-h686p" podStartSLOduration=1.488748835 podStartE2EDuration="4.763742347s" totalImagesPullingTime="3.274993512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:00:16.762461531 +0000 UTC m=+133.198516313" watchObservedRunningTime="2026-09-10 23:00:16.763742347 +0000 UTC m=+133.199797119"
Sep 10 23:00:16.778371 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.778326    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-fs2ps" podStartSLOduration=130.815153632 podStartE2EDuration="2m12.778312483s" totalImagesPullingTime="1.963158851s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 22:58:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:00:16.777655805 +0000 UTC m=+133.213710579" watchObservedRunningTime="2026-09-10 23:00:16.778312483 +0000 UTC m=+133.214367275"
Sep 10 23:00:16.883027 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.882940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:16.944312 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:16.944278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:17.192454 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:17.192419    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:17.313301 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:17.313259    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:20.760582 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:20.760534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:22.711910 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:22.711862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:22.712310 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:22.711961    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:22.716649 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:22.716625    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:22.771698 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:22.771666    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:24.756989 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:24.756951    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:24.858042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:24.858007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:26.758670 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:26.758624    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:26.858119 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:26.858082    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:27.132825 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:27.132694    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:27.188773 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:27.188740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:28.561254 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:28.561220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:28.666158 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:28.666124    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:31.561162 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:31.561128    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:31.633287 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:31.633254    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:32.111200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:32.111167    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58575c88cc-h686p"]
Sep 10 23:00:33.176260 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:33.176226    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:33.276909 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:33.276873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:34.572665 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:34.572634    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:34.658136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:34.658107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:35.762927 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:35.762892    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:46.525845 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:46.525804    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:46.594899 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:46.594866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:00:47.852662 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:47.852630    2577 generic.go:356] "Generic (PLEG): container finished" podID="9676f20c-a73c-4e02-97dd-9c12b6792649" containerID="caaec742cf61bf961dd560eaca1be1c4e81c6e73fa818919d3d50b6bd0ef154a" exitCode=0
Sep 10 23:00:47.853042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:47.852702    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m" event={"ID":"9676f20c-a73c-4e02-97dd-9c12b6792649","Type":"ContainerDied","Data":"caaec742cf61bf961dd560eaca1be1c4e81c6e73fa818919d3d50b6bd0ef154a"}
Sep 10 23:00:47.853042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:47.853008    2577 scope.go:117] "RemoveContainer" containerID="caaec742cf61bf961dd560eaca1be1c4e81c6e73fa818919d3d50b6bd0ef154a"
Sep 10 23:00:48.858275 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:48.858242    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6c8f84f897-9nj9m" event={"ID":"9676f20c-a73c-4e02-97dd-9c12b6792649","Type":"ContainerStarted","Data":"32a375907984ba38a05d04f902ff6ee31fdfc5d63ef8278df2267013a8d6fed3"}
Sep 10 23:00:57.130445 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.130330    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58575c88cc-h686p" podUID="3ae5649b-b75b-47c2-9635-37dae8f603e5" containerName="console" containerID="cri-o://2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b" gracePeriod=15
Sep 10 23:00:57.364285 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.364258    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:57.519991 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.519949    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-oauth-config\") pod \"3ae5649b-b75b-47c2-9635-37dae8f603e5\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") "
Sep 10 23:00:57.519991 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.519991    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-oauth-serving-cert\") pod \"3ae5649b-b75b-47c2-9635-37dae8f603e5\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") "
Sep 10 23:00:57.520253 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.520013    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-config\") pod \"3ae5649b-b75b-47c2-9635-37dae8f603e5\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") "
Sep 10 23:00:57.520253 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.520033    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jmdr\" (UniqueName: \"kubernetes.io/projected/3ae5649b-b75b-47c2-9635-37dae8f603e5-kube-api-access-7jmdr\") pod \"3ae5649b-b75b-47c2-9635-37dae8f603e5\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") "
Sep 10 23:00:57.520253 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.520075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-service-ca\") pod \"3ae5649b-b75b-47c2-9635-37dae8f603e5\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") "
Sep 10 23:00:57.520253 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.520104    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-serving-cert\") pod \"3ae5649b-b75b-47c2-9635-37dae8f603e5\" (UID: \"3ae5649b-b75b-47c2-9635-37dae8f603e5\") "
Sep 10 23:00:57.520542 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.520503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-config" (OuterVolumeSpecName: "console-config") pod "3ae5649b-b75b-47c2-9635-37dae8f603e5" (UID: "3ae5649b-b75b-47c2-9635-37dae8f603e5"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:00:57.520664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.520543    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-service-ca" (OuterVolumeSpecName: "service-ca") pod "3ae5649b-b75b-47c2-9635-37dae8f603e5" (UID: "3ae5649b-b75b-47c2-9635-37dae8f603e5"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:00:57.520664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.520541    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "3ae5649b-b75b-47c2-9635-37dae8f603e5" (UID: "3ae5649b-b75b-47c2-9635-37dae8f603e5"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:00:57.522370 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.522340    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "3ae5649b-b75b-47c2-9635-37dae8f603e5" (UID: "3ae5649b-b75b-47c2-9635-37dae8f603e5"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:00:57.522461 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.522392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "3ae5649b-b75b-47c2-9635-37dae8f603e5" (UID: "3ae5649b-b75b-47c2-9635-37dae8f603e5"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:00:57.522461 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.522402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ae5649b-b75b-47c2-9635-37dae8f603e5-kube-api-access-7jmdr" (OuterVolumeSpecName: "kube-api-access-7jmdr") pod "3ae5649b-b75b-47c2-9635-37dae8f603e5" (UID: "3ae5649b-b75b-47c2-9635-37dae8f603e5"). InnerVolumeSpecName "kube-api-access-7jmdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:00:57.621568 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.621527    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-oauth-config\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:57.621568 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.621561    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-oauth-serving-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:57.621568 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.621572    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-config\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:57.621822 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.621581    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jmdr\" (UniqueName: \"kubernetes.io/projected/3ae5649b-b75b-47c2-9635-37dae8f603e5-kube-api-access-7jmdr\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:57.621822 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.621589    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3ae5649b-b75b-47c2-9635-37dae8f603e5-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:57.621822 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.621597    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3ae5649b-b75b-47c2-9635-37dae8f603e5-console-serving-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:00:57.895280 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.895188    2577 generic.go:356] "Generic (PLEG): container finished" podID="3ae5649b-b75b-47c2-9635-37dae8f603e5" containerID="2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b" exitCode=0
Sep 10 23:00:57.895280 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.895230    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58575c88cc-h686p"
Sep 10 23:00:57.895280 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.895259    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58575c88cc-h686p" event={"ID":"3ae5649b-b75b-47c2-9635-37dae8f603e5","Type":"ContainerDied","Data":"2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b"}
Sep 10 23:00:57.895503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.895297    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58575c88cc-h686p" event={"ID":"3ae5649b-b75b-47c2-9635-37dae8f603e5","Type":"ContainerDied","Data":"06841dba423a37c7dfa6b70992ce3f801fed82524aa3ac5d200cd1514f5225be"}
Sep 10 23:00:57.895503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.895315    2577 scope.go:117] "RemoveContainer" containerID="2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b"
Sep 10 23:00:57.903586 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.903558    2577 scope.go:117] "RemoveContainer" containerID="2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b"
Sep 10 23:00:57.903890 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:00:57.903872    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b\": container with ID starting with 2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b not found: ID does not exist" containerID="2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b"
Sep 10 23:00:57.903946 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.903900    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b"} err="failed to get container status \"2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b\": rpc error: code = NotFound desc = could not find container \"2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b\": container with ID starting with 2da8d8200ba14f71f328eb3f08d5cb9a9e45732e8722994281508754b0b60d3b not found: ID does not exist"
Sep 10 23:00:57.916523 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.916492    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58575c88cc-h686p"]
Sep 10 23:00:57.926339 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:57.926312    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58575c88cc-h686p"]
Sep 10 23:00:58.213498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:00:58.213464    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3ae5649b-b75b-47c2-9635-37dae8f603e5" path="/var/lib/kubelet/pods/3ae5649b-b75b-47c2-9635-37dae8f603e5/volumes"
Sep 10 23:01:11.126528 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:11.126494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:14.806243 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:14.806204    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:15.110696 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:15.110615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:23.432797 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:23.432749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:23.489989 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:23.489947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:34.462102 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:34.462062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:34.563034 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:34.562998    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:01:34.621627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:01:34.621581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:10.797669 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:10.797636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:10.900761 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:10.900727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:30.090568 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:30.090483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:30.147115 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:30.147077    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:39.847613 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.847577    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7589d58688-cvlxf"]
Sep 10 23:02:39.848068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.847833    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3ae5649b-b75b-47c2-9635-37dae8f603e5" containerName="console"
Sep 10 23:02:39.848068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.847844    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3ae5649b-b75b-47c2-9635-37dae8f603e5" containerName="console"
Sep 10 23:02:39.848068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.847885    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3ae5649b-b75b-47c2-9635-37dae8f603e5" containerName="console"
Sep 10 23:02:39.850534 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.850518    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:39.852755 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.852733    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-29xcp\""
Sep 10 23:02:39.852905 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.852736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 23:02:39.852905 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.852736    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 23:02:39.852905 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.852736    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 23:02:39.853069 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.853054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 23:02:39.853176 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.853160    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 23:02:39.853220 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.853172    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 23:02:39.853273 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.853247    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 23:02:39.858271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.858233    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 23:02:39.860811 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.860771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7589d58688-cvlxf"]
Sep 10 23:02:39.911240 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.911205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:39.942297 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.942265    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-console-config\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:39.942410 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.942307    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-serving-cert\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:39.942410 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.942334    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-oauth-serving-cert\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:39.942410 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.942373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-service-ca\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:39.942410 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.942394    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkl2s\" (UniqueName: \"kubernetes.io/projected/9fffba91-5aef-4072-98e4-822e75e351e8-kube-api-access-zkl2s\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:39.942537 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.942425    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-oauth-config\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:39.942537 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:39.942463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-trusted-ca-bundle\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.043122 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.043082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-trusted-ca-bundle\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.043299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.043145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-console-config\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.043299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.043175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-serving-cert\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.043299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.043198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-oauth-serving-cert\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.043299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.043241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-service-ca\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.043299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.043275    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkl2s\" (UniqueName: \"kubernetes.io/projected/9fffba91-5aef-4072-98e4-822e75e351e8-kube-api-access-zkl2s\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.043495 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.043303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-oauth-config\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.044369 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.044337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-console-config\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.044874 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.044380    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-oauth-serving-cert\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.045004 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.044393    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-service-ca\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.045063 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.045033    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-trusted-ca-bundle\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.046619 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.046595    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:40.049806 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.048612    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-serving-cert\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.050687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.050662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-oauth-config\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.052881 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.052851    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkl2s\" (UniqueName: \"kubernetes.io/projected/9fffba91-5aef-4072-98e4-822e75e351e8-kube-api-access-zkl2s\") pod \"console-7589d58688-cvlxf\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") " pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.113053 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.112966    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:40.160132 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.160098    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:40.434291 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.434256    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7589d58688-cvlxf"]
Sep 10 23:02:40.437411 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:02:40.437381    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fffba91_5aef_4072_98e4_822e75e351e8.slice/crio-fd3369536c259986873616830165828dba086c28a21db79491217cd24c4dfe38 WatchSource:0}: Error finding container fd3369536c259986873616830165828dba086c28a21db79491217cd24c4dfe38: Status 404 returned error can't find the container with id fd3369536c259986873616830165828dba086c28a21db79491217cd24c4dfe38
Sep 10 23:02:40.646973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.646940    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:40.706702 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:40.706630    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:41.242669 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:41.242632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7589d58688-cvlxf" event={"ID":"9fffba91-5aef-4072-98e4-822e75e351e8","Type":"ContainerStarted","Data":"a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3"}
Sep 10 23:02:41.243189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:41.242682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7589d58688-cvlxf" event={"ID":"9fffba91-5aef-4072-98e4-822e75e351e8","Type":"ContainerStarted","Data":"fd3369536c259986873616830165828dba086c28a21db79491217cd24c4dfe38"}
Sep 10 23:02:41.265688 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:41.265633    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7589d58688-cvlxf" podStartSLOduration=2.265614701 podStartE2EDuration="2.265614701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:02:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:02:41.264217723 +0000 UTC m=+277.700272496" watchObservedRunningTime="2026-09-10 23:02:41.265614701 +0000 UTC m=+277.701669486"
Sep 10 23:02:41.983468 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:41.983433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:42.084649 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:42.084614    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:42.145616 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:42.145585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:46.529498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:46.529460    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:46.642240 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:46.642201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:46.739444 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:46.739404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:50.161248 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:50.161205    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:50.161669 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:50.161267    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:50.166143 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:50.166118    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:50.280464 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:50.280436    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:02:50.441466 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:50.441364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:50.501217 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:50.501187    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:02:50.558045 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:02:50.558012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:00.575474 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:00.575436    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:00.634258 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:00.634228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:03.997325 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:03.997291    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 23:03:21.019951 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:21.019916    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:30.863138 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:30.863101    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:30.959774 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:30.959737    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:42.886017 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:42.885983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:52.315662 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:52.315621    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:53.681202 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:53.681163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:55.936660 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:55.936615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:56.039591 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:56.039510    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:57.848798 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:57.848745    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:57.929867 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:57.929818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:58.228261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:58.228231    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:58.516984 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:58.516911    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:03:59.761917 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:03:59.761875    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:00.122295 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:00.122214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:01.288151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.288119    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"]
Sep 10 23:04:01.290988 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.290972    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.293388 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.293364    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 23:04:01.294217 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.294191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 23:04:01.294217 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.294206    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-xqbxk\""
Sep 10 23:04:01.298746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.298725    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"]
Sep 10 23:04:01.418888 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.418841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cw8p\" (UniqueName: \"kubernetes.io/projected/e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0-kube-api-access-5cw8p\") pod \"cert-manager-cainjector-559cdd8c87-wnct2\" (UID: \"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.418888 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.418890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wnct2\" (UID: \"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.519338 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.519304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cw8p\" (UniqueName: \"kubernetes.io/projected/e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0-kube-api-access-5cw8p\") pod \"cert-manager-cainjector-559cdd8c87-wnct2\" (UID: \"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.519338 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.519341    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wnct2\" (UID: \"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.532665 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.532632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-wnct2\" (UID: \"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.532822 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.532803    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cw8p\" (UniqueName: \"kubernetes.io/projected/e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0-kube-api-access-5cw8p\") pod \"cert-manager-cainjector-559cdd8c87-wnct2\" (UID: \"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.600505 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.600422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"
Sep 10 23:04:01.870131 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.870051    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-wnct2"]
Sep 10 23:04:01.875536 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:04:01.875501    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1bfef74_fcb4_482b_8cf9_2cfc22b3c5f0.slice/crio-664bb1e438c8110b5a87f91d02585a683f02ddc19b207d43348a46f468daa131 WatchSource:0}: Error finding container 664bb1e438c8110b5a87f91d02585a683f02ddc19b207d43348a46f468daa131: Status 404 returned error can't find the container with id 664bb1e438c8110b5a87f91d02585a683f02ddc19b207d43348a46f468daa131
Sep 10 23:04:01.877389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:01.877368    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:04:02.520440 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:02.520403    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2" event={"ID":"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0","Type":"ContainerStarted","Data":"664bb1e438c8110b5a87f91d02585a683f02ddc19b207d43348a46f468daa131"}
Sep 10 23:04:02.922406 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:02.922356    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:03.023453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:03.023394    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:05.535719 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:05.535678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2" event={"ID":"e1bfef74-fcb4-482b-8cf9-2cfc22b3c5f0","Type":"ContainerStarted","Data":"141d25563a88d2ab166ebcbef7a82fa15d54a9d6a53fa13bea25921753310ec5"}
Sep 10 23:04:05.554887 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:05.554830    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-wnct2" podStartSLOduration=1.5477221330000002 podStartE2EDuration="4.554808727s" totalImagesPullingTime="3.007086594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:04:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:04:05.554555343 +0000 UTC m=+361.990610115" watchObservedRunningTime="2026-09-10 23:04:05.554808727 +0000 UTC m=+361.990863495"
Sep 10 23:04:06.499535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:06.499482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:06.570580 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:06.570547    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:08.523474 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:08.523438    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:10.602244 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:10.602213    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:10.704449 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:10.704417    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:15.577698 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:15.577662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:15.682243 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:15.682205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:17.520083 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:17.520050    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:25.771134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:25.771097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:25.862523 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:25.862487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:32.562129 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:32.562095    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:32.663925 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:32.663887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:33.152542 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.152509    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:33.509631 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.509544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:33.598725 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.598687    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:33.632566 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.632525    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"]
Sep 10 23:04:33.635736 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.635715    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.637912 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.637885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 23:04:33.638052 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.637948    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dppfd\""
Sep 10 23:04:33.638052 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.637998    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 23:04:33.638052 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.638024    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 23:04:33.638449 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.638425    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 23:04:33.653821 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.653797    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"]
Sep 10 23:04:33.778844 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.778726    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8349091b-3589-443f-961f-23d0a6155a5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.778844 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.778801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jslb\" (UniqueName: \"kubernetes.io/projected/8349091b-3589-443f-961f-23d0a6155a5b-kube-api-access-2jslb\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.778844 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.778833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8349091b-3589-443f-961f-23d0a6155a5b-tmp\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.779104 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.778866    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8349091b-3589-443f-961f-23d0a6155a5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.779104 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.778927    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8349091b-3589-443f-961f-23d0a6155a5b-manifests\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.802031 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.801996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:33.879764 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.879723    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8349091b-3589-443f-961f-23d0a6155a5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.879962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.879799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jslb\" (UniqueName: \"kubernetes.io/projected/8349091b-3589-443f-961f-23d0a6155a5b-kube-api-access-2jslb\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.879962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.879829    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8349091b-3589-443f-961f-23d0a6155a5b-tmp\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.879962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.879860    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8349091b-3589-443f-961f-23d0a6155a5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.879962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.879902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8349091b-3589-443f-961f-23d0a6155a5b-manifests\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.880368 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.880349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/8349091b-3589-443f-961f-23d0a6155a5b-manifests\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.882157 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.882134    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/8349091b-3589-443f-961f-23d0a6155a5b-tmp\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.882508 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.882485    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8349091b-3589-443f-961f-23d0a6155a5b-webhook-cert\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.882549 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.882519    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/8349091b-3589-443f-961f-23d0a6155a5b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.893257 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.893225    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jslb\" (UniqueName: \"kubernetes.io/projected/8349091b-3589-443f-961f-23d0a6155a5b-kube-api-access-2jslb\") pod \"opendatahub-operator-controller-manager-dbfbc5c4d-96z5n\" (UID: \"8349091b-3589-443f-961f-23d0a6155a5b\") " pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:33.906111 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.906062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:33.945962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:33.945931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:34.006561 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.006505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.110526 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.110448    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.203826 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.203799    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.230945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.230917    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"]
Sep 10 23:04:34.233078 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:04:34.233054    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8349091b_3589_443f_961f_23d0a6155a5b.slice/crio-c0a6a9cbd10110747ada9b233b2035b8cff06b9519ad8046fbda69f87dfcbe01 WatchSource:0}: Error finding container c0a6a9cbd10110747ada9b233b2035b8cff06b9519ad8046fbda69f87dfcbe01: Status 404 returned error can't find the container with id c0a6a9cbd10110747ada9b233b2035b8cff06b9519ad8046fbda69f87dfcbe01
Sep 10 23:04:34.305432 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.305385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.404640 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.404604    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.505107 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.505073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:34.642499 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:34.642463    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n" event={"ID":"8349091b-3589-443f-961f-23d0a6155a5b","Type":"ContainerStarted","Data":"c0a6a9cbd10110747ada9b233b2035b8cff06b9519ad8046fbda69f87dfcbe01"}
Sep 10 23:04:37.656302 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:37.656264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n" event={"ID":"8349091b-3589-443f-961f-23d0a6155a5b","Type":"ContainerStarted","Data":"1df18d3a3afbda728ab27961e74ef10daeb4da5d754449b8932386cd0b03e0b2"}
Sep 10 23:04:38.661697 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:38.661662    2577 generic.go:356] "Generic (PLEG): container finished" podID="8349091b-3589-443f-961f-23d0a6155a5b" containerID="1df18d3a3afbda728ab27961e74ef10daeb4da5d754449b8932386cd0b03e0b2" exitCode=0
Sep 10 23:04:38.662089 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:38.661703    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n" event={"ID":"8349091b-3589-443f-961f-23d0a6155a5b","Type":"ContainerDied","Data":"1df18d3a3afbda728ab27961e74ef10daeb4da5d754449b8932386cd0b03e0b2"}
Sep 10 23:04:39.667668 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:39.667629    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n" event={"ID":"8349091b-3589-443f-961f-23d0a6155a5b","Type":"ContainerStarted","Data":"1a587a4c7bcf97916a6ef3459ab8853ed5ccfe83d2cfc3149c835d58097d08eb"}
Sep 10 23:04:39.668159 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:39.667748    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:39.693152 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:39.693094    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n" podStartSLOduration=1.759598555 podStartE2EDuration="6.693076968s" totalImagesPullingTime="3.91624961s" totalInitContainerRuntime="1.017228803s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:04:33 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:04:39.691962076 +0000 UTC m=+396.128016849" watchObservedRunningTime="2026-09-10 23:04:39.693076968 +0000 UTC m=+396.129131740"
Sep 10 23:04:50.672948 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:50.672912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-dbfbc5c4d-96z5n"
Sep 10 23:04:53.814437 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:53.814404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:53.938916 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:53.938880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:59.414337 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.414304    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:04:59.467486 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.467446    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-7d9b555765-f2852"]
Sep 10 23:04:59.470641 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.470614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.473515 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.473480    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 23:04:59.473515 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.473486    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 23:04:59.473709 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.473529    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 23:04:59.473816 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.473745    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 23:04:59.474392 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.474377    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-n4bs2\""
Sep 10 23:04:59.489830 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.488746    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d9b555765-f2852"]
Sep 10 23:04:59.566775 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.566731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6k8q\" (UniqueName: \"kubernetes.io/projected/3a6f77ac-e13d-4a61-8f72-32128f0330d2-kube-api-access-z6k8q\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.566973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.566809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tls-certs\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.566973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.566855    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tmp\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.667658 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.667559    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tls-certs\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.667658 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.667614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tmp\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.667937 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.667666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6k8q\" (UniqueName: \"kubernetes.io/projected/3a6f77ac-e13d-4a61-8f72-32128f0330d2-kube-api-access-z6k8q\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.667937 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:04:59.667711    2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 23:04:59.667937 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:04:59.667821    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tls-certs podName:3a6f77ac-e13d-4a61-8f72-32128f0330d2 nodeName:}" failed. No retries permitted until 2026-09-10 23:05:00.167800942 +0000 UTC m=+416.603855716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tls-certs") pod "kube-auth-proxy-7d9b555765-f2852" (UID: "3a6f77ac-e13d-4a61-8f72-32128f0330d2") : secret "kube-auth-proxy-tls" not found
Sep 10 23:04:59.669933 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.669911    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tmp\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.679396 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.679371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6k8q\" (UniqueName: \"kubernetes.io/projected/3a6f77ac-e13d-4a61-8f72-32128f0330d2-kube-api-access-z6k8q\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:04:59.876136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:04:59.876099    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:00.173049 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.173012    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tls-certs\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:05:00.175405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.175379    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a6f77ac-e13d-4a61-8f72-32128f0330d2-tls-certs\") pod \"kube-auth-proxy-7d9b555765-f2852\" (UID: \"3a6f77ac-e13d-4a61-8f72-32128f0330d2\") " pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:05:00.382262 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.382230    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852"
Sep 10 23:05:00.660937 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.660904    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-7d9b555765-f2852"]
Sep 10 23:05:00.664851 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:05:00.664823    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a6f77ac_e13d_4a61_8f72_32128f0330d2.slice/crio-4629f335572385b1a19db30d5b3edbe0903387d79137f97f928610c2bbb99657 WatchSource:0}: Error finding container 4629f335572385b1a19db30d5b3edbe0903387d79137f97f928610c2bbb99657: Status 404 returned error can't find the container with id 4629f335572385b1a19db30d5b3edbe0903387d79137f97f928610c2bbb99657
Sep 10 23:05:00.741745 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.741651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852" event={"ID":"3a6f77ac-e13d-4a61-8f72-32128f0330d2","Type":"ContainerStarted","Data":"4629f335572385b1a19db30d5b3edbe0903387d79137f97f928610c2bbb99657"}
Sep 10 23:05:00.774647 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.774620    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"]
Sep 10 23:05:00.777027 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.776999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:00.778817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.778801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.781159 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.781139    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-bfwbc\""
Sep 10 23:05:00.781249 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.781174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 23:05:00.781249 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.781198    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 23:05:00.781512 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.781494    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 23:05:00.781589 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.781531    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 23:05:00.791141 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.791117    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"]
Sep 10 23:05:00.850629 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.850600    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:00.878554 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.878526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.878680 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.878557    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqvgv\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-kube-api-access-lqvgv\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.878680 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.878600    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.878680 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.878618    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.878680 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.878636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.878860 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.878693    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.878860 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.878718    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.979239 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.979201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.979380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.979248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.979380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.979345    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.979380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.979363    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqvgv\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-kube-api-access-lqvgv\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.979507 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.979399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.979507 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.979419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.979507 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.979436    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.980071 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.980042    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.981705 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.981678    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.981821 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.981747    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.982085 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.982068    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.982140 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.982121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.988530 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.988495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqvgv\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-kube-api-access-lqvgv\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:00.988656 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:00.988603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-g9w2m\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:01.087816 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:01.087677    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:01.364297 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:01.364218    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"]
Sep 10 23:05:01.367115 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:05:01.367088    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07104f31_e132_4a2c_a7a5_bac9fb9ae8df.slice/crio-788b88d0614a8df9945598ff28c0201ffa8419a60783a1314ab1fe0c37433dde WatchSource:0}: Error finding container 788b88d0614a8df9945598ff28c0201ffa8419a60783a1314ab1fe0c37433dde: Status 404 returned error can't find the container with id 788b88d0614a8df9945598ff28c0201ffa8419a60783a1314ab1fe0c37433dde
Sep 10 23:05:01.748967 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:01.748922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m" event={"ID":"07104f31-e132-4a2c-a7a5-bac9fb9ae8df","Type":"ContainerStarted","Data":"788b88d0614a8df9945598ff28c0201ffa8419a60783a1314ab1fe0c37433dde"}
Sep 10 23:05:01.961908 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:01.961871    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:03.761048 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:03.760956    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852" event={"ID":"3a6f77ac-e13d-4a61-8f72-32128f0330d2","Type":"ContainerStarted","Data":"ddd2031037c860844d4ce857708b33e9bc5c0a86e11d4aeeb3629d051f59ebe8"}
Sep 10 23:05:03.779090 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:03.778992    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-7d9b555765-f2852" podStartSLOduration=2.028626828 podStartE2EDuration="4.778970785s" totalImagesPullingTime="2.750343957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:04:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:03.778625286 +0000 UTC m=+420.214680063" watchObservedRunningTime="2026-09-10 23:05:03.778970785 +0000 UTC m=+420.215025561"
Sep 10 23:05:04.786935 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:04.786900    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:04.881901 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:04.881853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:04.984522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:04.984481    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:05.217641 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.217591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:05.261530 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.261476    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:05:05.261637 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.261547    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:05:05.357518 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.356956    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qtwzt"]
Sep 10 23:05:05.360361 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.360339    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:05.362979 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.362952    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 10 23:05:05.372744 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.372718    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qtwzt"]
Sep 10 23:05:05.377663 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.377636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:05.420039 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.419996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd2fa529-5ccd-4881-b548-484513d752e5-tls-certs\") pod \"model-serving-api-54d89bd5fc-qtwzt\" (UID: \"dd2fa529-5ccd-4881-b548-484513d752e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:05.420201 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.420052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4268w\" (UniqueName: \"kubernetes.io/projected/dd2fa529-5ccd-4881-b548-484513d752e5-kube-api-access-4268w\") pod \"model-serving-api-54d89bd5fc-qtwzt\" (UID: \"dd2fa529-5ccd-4881-b548-484513d752e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:05.521099 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.520993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4268w\" (UniqueName: \"kubernetes.io/projected/dd2fa529-5ccd-4881-b548-484513d752e5-kube-api-access-4268w\") pod \"model-serving-api-54d89bd5fc-qtwzt\" (UID: \"dd2fa529-5ccd-4881-b548-484513d752e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:05.521099 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.521072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd2fa529-5ccd-4881-b548-484513d752e5-tls-certs\") pod \"model-serving-api-54d89bd5fc-qtwzt\" (UID: \"dd2fa529-5ccd-4881-b548-484513d752e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:05.521340 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:05:05.521165    2577 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Sep 10 23:05:05.521340 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:05:05.521221    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd2fa529-5ccd-4881-b548-484513d752e5-tls-certs podName:dd2fa529-5ccd-4881-b548-484513d752e5 nodeName:}" failed. No retries permitted until 2026-09-10 23:05:06.02120386 +0000 UTC m=+422.457258611 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/dd2fa529-5ccd-4881-b548-484513d752e5-tls-certs") pod "model-serving-api-54d89bd5fc-qtwzt" (UID: "dd2fa529-5ccd-4881-b548-484513d752e5") : secret "model-serving-api-tls" not found
Sep 10 23:05:05.533200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.533165    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4268w\" (UniqueName: \"kubernetes.io/projected/dd2fa529-5ccd-4881-b548-484513d752e5-kube-api-access-4268w\") pod \"model-serving-api-54d89bd5fc-qtwzt\" (UID: \"dd2fa529-5ccd-4881-b548-484513d752e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:05.615468 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.615435    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:05.774356 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.774256    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m" event={"ID":"07104f31-e132-4a2c-a7a5-bac9fb9ae8df","Type":"ContainerStarted","Data":"ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d"}
Sep 10 23:05:05.774513 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.774427    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:05.799097 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.799035    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m" podStartSLOduration=1.906964514 podStartE2EDuration="5.799010827s" totalImagesPullingTime="3.892046313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:05.796641004 +0000 UTC m=+422.232695777" watchObservedRunningTime="2026-09-10 23:05:05.799010827 +0000 UTC m=+422.235065601"
Sep 10 23:05:05.818933 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.818887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:05.923347 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:05.923315    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:06.025820 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:06.025722    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd2fa529-5ccd-4881-b548-484513d752e5-tls-certs\") pod \"model-serving-api-54d89bd5fc-qtwzt\" (UID: \"dd2fa529-5ccd-4881-b548-484513d752e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:06.028221 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:06.028198    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dd2fa529-5ccd-4881-b548-484513d752e5-tls-certs\") pod \"model-serving-api-54d89bd5fc-qtwzt\" (UID: \"dd2fa529-5ccd-4881-b548-484513d752e5\") " pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:06.271125 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:06.271093    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:06.545633 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:06.545423    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-qtwzt"]
Sep 10 23:05:06.548432 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:05:06.548405    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd2fa529_5ccd_4881_b548_484513d752e5.slice/crio-4cc4f578e8e155747123746b203e0a53f0c968dc48e546fd8be6006619d41d75 WatchSource:0}: Error finding container 4cc4f578e8e155747123746b203e0a53f0c968dc48e546fd8be6006619d41d75: Status 404 returned error can't find the container with id 4cc4f578e8e155747123746b203e0a53f0c968dc48e546fd8be6006619d41d75
Sep 10 23:05:06.779366 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:06.779327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt" event={"ID":"dd2fa529-5ccd-4881-b548-484513d752e5","Type":"ContainerStarted","Data":"4cc4f578e8e155747123746b203e0a53f0c968dc48e546fd8be6006619d41d75"}
Sep 10 23:05:06.781328 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:06.781303    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:06.861665 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:06.861582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:07.551269 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:07.551235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:07.667449 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:07.667420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:08.788670 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:08.788632    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt" event={"ID":"dd2fa529-5ccd-4881-b548-484513d752e5","Type":"ContainerStarted","Data":"3bf434db2164855094da9a467a6c7ab542deee8327848b371d558a4e8c37484d"}
Sep 10 23:05:08.789174 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:08.788760    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:08.811213 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:08.811168    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt" podStartSLOduration=2.208578088 podStartE2EDuration="3.811150167s" totalImagesPullingTime="1.602572079s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:08.810061446 +0000 UTC m=+425.246116220" watchObservedRunningTime="2026-09-10 23:05:08.811150167 +0000 UTC m=+425.247204941"
Sep 10 23:05:11.925945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.925905    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"]
Sep 10 23:05:11.934110 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.934079    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:11.945721 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.945694    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"]
Sep 10 23:05:11.976061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.976022    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2hp\" (UniqueName: \"kubernetes.io/projected/5c8ba649-824d-442b-ad3a-2113c060dca6-kube-api-access-bf2hp\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:11.976261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.976109    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:11.976261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.976162    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8ba649-824d-442b-ad3a-2113c060dca6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:11.976261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.976204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:11.976417 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.976261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:11.976417 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.976314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:11.976417 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:11.976346    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.077802 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.077738    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2hp\" (UniqueName: \"kubernetes.io/projected/5c8ba649-824d-442b-ad3a-2113c060dca6-kube-api-access-bf2hp\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.077802 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.077803    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.078061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.077831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8ba649-824d-442b-ad3a-2113c060dca6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.078061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.077855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.078061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.077884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.078061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.077997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.078061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.078035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.078693 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.078662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.080365 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.080333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c8ba649-824d-442b-ad3a-2113c060dca6-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.080365 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.080361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.080503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.080464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.080503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.080477    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.091774 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.091751    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2hp\" (UniqueName: \"kubernetes.io/projected/5c8ba649-824d-442b-ad3a-2113c060dca6-kube-api-access-bf2hp\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.092203 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.092187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c8ba649-824d-442b-ad3a-2113c060dca6-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-jv9xt\" (UID: \"5c8ba649-824d-442b-ad3a-2113c060dca6\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.243942 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.243847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.566207 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.566179    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"]
Sep 10 23:05:12.568381 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:05:12.568354    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c8ba649_824d_442b_ad3a_2113c060dca6.slice/crio-43dec1a5db5692aa366cb10aaf5204be04afcc624bbb3b657d1883ac7e40fd35 WatchSource:0}: Error finding container 43dec1a5db5692aa366cb10aaf5204be04afcc624bbb3b657d1883ac7e40fd35: Status 404 returned error can't find the container with id 43dec1a5db5692aa366cb10aaf5204be04afcc624bbb3b657d1883ac7e40fd35
Sep 10 23:05:12.571408 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.571380    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:05:12.571472 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.571437    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236212Ki","pods":"250"}
Sep 10 23:05:12.807696 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.807660    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt" event={"ID":"5c8ba649-824d-442b-ad3a-2113c060dca6","Type":"ContainerStarted","Data":"6d47f8b73a8ac9c71ab84e11da06685e18bd49cba33b43a7f16ac79bd5e1a4f6"}
Sep 10 23:05:12.807907 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.807715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt" event={"ID":"5c8ba649-824d-442b-ad3a-2113c060dca6","Type":"ContainerStarted","Data":"43dec1a5db5692aa366cb10aaf5204be04afcc624bbb3b657d1883ac7e40fd35"}
Sep 10 23:05:12.807907 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.807744    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:12.836586 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.836472    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt" podStartSLOduration=1.836453738 podStartE2EDuration="1.836453738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:12.835176685 +0000 UTC m=+429.271231459" watchObservedRunningTime="2026-09-10 23:05:12.836453738 +0000 UTC m=+429.272508510"
Sep 10 23:05:12.837727 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.837705    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:12.907498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:12.907465    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:13.813612 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:13.813585    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-jv9xt"
Sep 10 23:05:13.894636 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:13.894595    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"]
Sep 10 23:05:13.894847 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:13.894820    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m" podUID="07104f31-e132-4a2c-a7a5-bac9fb9ae8df" containerName="discovery" containerID="cri-o://ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d" gracePeriod=30
Sep 10 23:05:14.146957 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.146930    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:14.199285 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199252    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-ca-configmap\") pod \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") "
Sep 10 23:05:14.199285 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-local-certs\") pod \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") "
Sep 10 23:05:14.199535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-token\") pod \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") "
Sep 10 23:05:14.199535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199348    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-dns-cert\") pod \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") "
Sep 10 23:05:14.199535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199384    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-kubeconfig\") pod \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") "
Sep 10 23:05:14.199535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqvgv\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-kube-api-access-lqvgv\") pod \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") "
Sep 10 23:05:14.199535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199439    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-cacerts\") pod \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\" (UID: \"07104f31-e132-4a2c-a7a5-bac9fb9ae8df\") "
Sep 10 23:05:14.199830 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.199642    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "07104f31-e132-4a2c-a7a5-bac9fb9ae8df" (UID: "07104f31-e132-4a2c-a7a5-bac9fb9ae8df"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:14.202071 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.202040    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "07104f31-e132-4a2c-a7a5-bac9fb9ae8df" (UID: "07104f31-e132-4a2c-a7a5-bac9fb9ae8df"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:05:14.202213 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.202172    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-token" (OuterVolumeSpecName: "istio-token") pod "07104f31-e132-4a2c-a7a5-bac9fb9ae8df" (UID: "07104f31-e132-4a2c-a7a5-bac9fb9ae8df"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:05:14.202276 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.202214    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-kube-api-access-lqvgv" (OuterVolumeSpecName: "kube-api-access-lqvgv") pod "07104f31-e132-4a2c-a7a5-bac9fb9ae8df" (UID: "07104f31-e132-4a2c-a7a5-bac9fb9ae8df"). InnerVolumeSpecName "kube-api-access-lqvgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:05:14.202276 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.202241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-local-certs" (OuterVolumeSpecName: "local-certs") pod "07104f31-e132-4a2c-a7a5-bac9fb9ae8df" (UID: "07104f31-e132-4a2c-a7a5-bac9fb9ae8df"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:05:14.202442 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.202410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-cacerts" (OuterVolumeSpecName: "cacerts") pod "07104f31-e132-4a2c-a7a5-bac9fb9ae8df" (UID: "07104f31-e132-4a2c-a7a5-bac9fb9ae8df"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:05:14.202564 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.202553    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "07104f31-e132-4a2c-a7a5-bac9fb9ae8df" (UID: "07104f31-e132-4a2c-a7a5-bac9fb9ae8df"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:05:14.300172 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.300138    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-kubeconfig\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:14.300172 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.300166    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqvgv\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-kube-api-access-lqvgv\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:14.300172 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.300176    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-cacerts\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:14.300408 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.300186    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-ca-configmap\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:14.300408 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.300196    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-local-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:14.300408 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.300205    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-token\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:14.300408 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.300212    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/07104f31-e132-4a2c-a7a5-bac9fb9ae8df-istio-csr-dns-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:14.816281 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.816243    2577 generic.go:356] "Generic (PLEG): container finished" podID="07104f31-e132-4a2c-a7a5-bac9fb9ae8df" containerID="ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d" exitCode=0
Sep 10 23:05:14.816690 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.816295    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"
Sep 10 23:05:14.816690 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.816327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m" event={"ID":"07104f31-e132-4a2c-a7a5-bac9fb9ae8df","Type":"ContainerDied","Data":"ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d"}
Sep 10 23:05:14.816690 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.816384    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m" event={"ID":"07104f31-e132-4a2c-a7a5-bac9fb9ae8df","Type":"ContainerDied","Data":"788b88d0614a8df9945598ff28c0201ffa8419a60783a1314ab1fe0c37433dde"}
Sep 10 23:05:14.816690 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.816405    2577 scope.go:117] "RemoveContainer" containerID="ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d"
Sep 10 23:05:14.824704 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.824689    2577 scope.go:117] "RemoveContainer" containerID="ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d"
Sep 10 23:05:14.825000 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:05:14.824978    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d\": container with ID starting with ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d not found: ID does not exist" containerID="ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d"
Sep 10 23:05:14.825083 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.825012    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d"} err="failed to get container status \"ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d\": rpc error: code = NotFound desc = could not find container \"ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d\": container with ID starting with ce3a3c0fcda56798e1667d0dd81f5a7a10dd693851edae72660abf03e3d0c65d not found: ID does not exist"
Sep 10 23:05:14.843094 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.843063    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"]
Sep 10 23:05:14.849299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.849276    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-g9w2m"]
Sep 10 23:05:14.955841 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:14.955808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:15.076305 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.076228    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"]
Sep 10 23:05:15.076517 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.076506    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07104f31-e132-4a2c-a7a5-bac9fb9ae8df" containerName="discovery"
Sep 10 23:05:15.076567 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.076518    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="07104f31-e132-4a2c-a7a5-bac9fb9ae8df" containerName="discovery"
Sep 10 23:05:15.076603 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.076583    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="07104f31-e132-4a2c-a7a5-bac9fb9ae8df" containerName="discovery"
Sep 10 23:05:15.079045 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.079029    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.081701 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.081675    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 23:05:15.081868 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.081680    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-j5jbr\""
Sep 10 23:05:15.082153 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.082130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:15.103152 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.103123    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"]
Sep 10 23:05:15.178291 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.178250    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:15.208351 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.208317    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnrhk\" (UniqueName: \"kubernetes.io/projected/50c25cd5-8dc4-4321-86e9-9d2a4a07bef5-kube-api-access-rnrhk\") pod \"kserve-controller-manager-d6f4b65b8-5mt9l\" (UID: \"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.208504 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.208373    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50c25cd5-8dc4-4321-86e9-9d2a4a07bef5-cert\") pod \"kserve-controller-manager-d6f4b65b8-5mt9l\" (UID: \"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.256606 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.256576    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:15.309747 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.309709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnrhk\" (UniqueName: \"kubernetes.io/projected/50c25cd5-8dc4-4321-86e9-9d2a4a07bef5-kube-api-access-rnrhk\") pod \"kserve-controller-manager-d6f4b65b8-5mt9l\" (UID: \"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.309942 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.309789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50c25cd5-8dc4-4321-86e9-9d2a4a07bef5-cert\") pod \"kserve-controller-manager-d6f4b65b8-5mt9l\" (UID: \"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.312602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.312563    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/50c25cd5-8dc4-4321-86e9-9d2a4a07bef5-cert\") pod \"kserve-controller-manager-d6f4b65b8-5mt9l\" (UID: \"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.324153 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.324124    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnrhk\" (UniqueName: \"kubernetes.io/projected/50c25cd5-8dc4-4321-86e9-9d2a4a07bef5-kube-api-access-rnrhk\") pod \"kserve-controller-manager-d6f4b65b8-5mt9l\" (UID: \"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.389179 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.389077    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:15.635444 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.635404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:15.664523 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.664492    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"]
Sep 10 23:05:15.667428 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:05:15.667401    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50c25cd5_8dc4_4321_86e9_9d2a4a07bef5.slice/crio-49ec457327d456772ce3214089a5ec800f181336cdf77818dfc9d8bb66b3f04a WatchSource:0}: Error finding container 49ec457327d456772ce3214089a5ec800f181336cdf77818dfc9d8bb66b3f04a: Status 404 returned error can't find the container with id 49ec457327d456772ce3214089a5ec800f181336cdf77818dfc9d8bb66b3f04a
Sep 10 23:05:15.823893 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:15.823854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l" event={"ID":"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5","Type":"ContainerStarted","Data":"49ec457327d456772ce3214089a5ec800f181336cdf77818dfc9d8bb66b3f04a"}
Sep 10 23:05:16.214289 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:16.214257    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07104f31-e132-4a2c-a7a5-bac9fb9ae8df" path="/var/lib/kubelet/pods/07104f31-e132-4a2c-a7a5-bac9fb9ae8df/volumes"
Sep 10 23:05:16.337729 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:16.337690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:16.451005 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:16.450963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:16.559306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:16.559218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:18.838944 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:18.838906    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l" event={"ID":"50c25cd5-8dc4-4321-86e9-9d2a4a07bef5","Type":"ContainerStarted","Data":"6275cd63fb81451fc598357c5a539f2d0f9f9b4863193d0149bceb64b14e7de1"}
Sep 10 23:05:18.839414 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:18.839029    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:18.869294 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:18.869243    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l" podStartSLOduration=1.711004462 podStartE2EDuration="3.86922728s" totalImagesPullingTime="2.158222818s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:05:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:05:18.866964156 +0000 UTC m=+435.303018929" watchObservedRunningTime="2026-09-10 23:05:18.86922728 +0000 UTC m=+435.305282053"
Sep 10 23:05:19.797066 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:19.797039    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-qtwzt"
Sep 10 23:05:23.378693 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:23.378652    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7589d58688-cvlxf"]
Sep 10 23:05:23.499645 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:23.499612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:23.606822 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:23.606762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:23.700360 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:23.700320    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:31.733240 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:31.733159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:31.797167 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:31.797139    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:05:48.398013 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.397967    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7589d58688-cvlxf" podUID="9fffba91-5aef-4072-98e4-822e75e351e8" containerName="console" containerID="cri-o://a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3" gracePeriod=15
Sep 10 23:05:48.634950 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.634923    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:05:48.785215 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-oauth-serving-cert\") pod \"9fffba91-5aef-4072-98e4-822e75e351e8\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") "
Sep 10 23:05:48.785405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-console-config\") pod \"9fffba91-5aef-4072-98e4-822e75e351e8\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") "
Sep 10 23:05:48.785405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785248    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-oauth-config\") pod \"9fffba91-5aef-4072-98e4-822e75e351e8\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") "
Sep 10 23:05:48.785405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-service-ca\") pod \"9fffba91-5aef-4072-98e4-822e75e351e8\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") "
Sep 10 23:05:48.785405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785307    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-trusted-ca-bundle\") pod \"9fffba91-5aef-4072-98e4-822e75e351e8\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") "
Sep 10 23:05:48.785405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785329    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-serving-cert\") pod \"9fffba91-5aef-4072-98e4-822e75e351e8\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") "
Sep 10 23:05:48.785405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785367    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zkl2s\" (UniqueName: \"kubernetes.io/projected/9fffba91-5aef-4072-98e4-822e75e351e8-kube-api-access-zkl2s\") pod \"9fffba91-5aef-4072-98e4-822e75e351e8\" (UID: \"9fffba91-5aef-4072-98e4-822e75e351e8\") "
Sep 10 23:05:48.785747 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785720    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9fffba91-5aef-4072-98e4-822e75e351e8" (UID: "9fffba91-5aef-4072-98e4-822e75e351e8"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:48.785833 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785746    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-console-config" (OuterVolumeSpecName: "console-config") pod "9fffba91-5aef-4072-98e4-822e75e351e8" (UID: "9fffba91-5aef-4072-98e4-822e75e351e8"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:48.785896 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785860    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-service-ca" (OuterVolumeSpecName: "service-ca") pod "9fffba91-5aef-4072-98e4-822e75e351e8" (UID: "9fffba91-5aef-4072-98e4-822e75e351e8"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:48.785933 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.785899    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9fffba91-5aef-4072-98e4-822e75e351e8" (UID: "9fffba91-5aef-4072-98e4-822e75e351e8"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:05:48.787592 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.787558    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9fffba91-5aef-4072-98e4-822e75e351e8" (UID: "9fffba91-5aef-4072-98e4-822e75e351e8"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:05:48.787987 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.787964    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fffba91-5aef-4072-98e4-822e75e351e8-kube-api-access-zkl2s" (OuterVolumeSpecName: "kube-api-access-zkl2s") pod "9fffba91-5aef-4072-98e4-822e75e351e8" (UID: "9fffba91-5aef-4072-98e4-822e75e351e8"). InnerVolumeSpecName "kube-api-access-zkl2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:05:48.787987 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.787965    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9fffba91-5aef-4072-98e4-822e75e351e8" (UID: "9fffba91-5aef-4072-98e4-822e75e351e8"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:05:48.886539 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.886499    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkl2s\" (UniqueName: \"kubernetes.io/projected/9fffba91-5aef-4072-98e4-822e75e351e8-kube-api-access-zkl2s\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:48.886539 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.886531    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-oauth-serving-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:48.886539 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.886541    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-console-config\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:48.886808 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.886549    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-oauth-config\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:48.886808 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.886558    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:48.886808 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.886566    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9fffba91-5aef-4072-98e4-822e75e351e8-trusted-ca-bundle\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:48.886808 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.886574    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9fffba91-5aef-4072-98e4-822e75e351e8-console-serving-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:05:48.961458 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.961421    2577 generic.go:356] "Generic (PLEG): container finished" podID="9fffba91-5aef-4072-98e4-822e75e351e8" containerID="a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3" exitCode=0
Sep 10 23:05:48.961610 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.961468    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7589d58688-cvlxf"
Sep 10 23:05:48.961610 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.961466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7589d58688-cvlxf" event={"ID":"9fffba91-5aef-4072-98e4-822e75e351e8","Type":"ContainerDied","Data":"a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3"}
Sep 10 23:05:48.961610 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.961592    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7589d58688-cvlxf" event={"ID":"9fffba91-5aef-4072-98e4-822e75e351e8","Type":"ContainerDied","Data":"fd3369536c259986873616830165828dba086c28a21db79491217cd24c4dfe38"}
Sep 10 23:05:48.961716 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.961619    2577 scope.go:117] "RemoveContainer" containerID="a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3"
Sep 10 23:05:48.970178 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.970160    2577 scope.go:117] "RemoveContainer" containerID="a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3"
Sep 10 23:05:48.970442 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:05:48.970421    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3\": container with ID starting with a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3 not found: ID does not exist" containerID="a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3"
Sep 10 23:05:48.970483 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.970451    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3"} err="failed to get container status \"a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3\": rpc error: code = NotFound desc = could not find container \"a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3\": container with ID starting with a48a1bbd8f0609e6cda728fc93f33db99b057682b547499aa52a9c85c9e669f3 not found: ID does not exist"
Sep 10 23:05:48.994208 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:48.994182    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7589d58688-cvlxf"]
Sep 10 23:05:49.004385 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:49.004361    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7589d58688-cvlxf"]
Sep 10 23:05:49.848110 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:49.848071    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5mt9l"
Sep 10 23:05:50.214219 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:50.214187    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9fffba91-5aef-4072-98e4-822e75e351e8" path="/var/lib/kubelet/pods/9fffba91-5aef-4072-98e4-822e75e351e8/volumes"
Sep 10 23:05:58.503380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:05:58.503344    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:05.922967 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:05.922932    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:06.013631 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:06.013598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:19.275426 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:19.275388    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:20.107113 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:20.107080    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:20.916887 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:20.916854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:23.369361 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.369334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:23.509985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.509939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:23.975161 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.975117    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"]
Sep 10 23:06:23.975435 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.975423    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9fffba91-5aef-4072-98e4-822e75e351e8" containerName="console"
Sep 10 23:06:23.975488 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.975436    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9fffba91-5aef-4072-98e4-822e75e351e8" containerName="console"
Sep 10 23:06:23.975530 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.975498    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9fffba91-5aef-4072-98e4-822e75e351e8" containerName="console"
Sep 10 23:06:23.977900 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.977884    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:23.980368 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.980342    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 23:06:23.980487 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.980399    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-264xh\""
Sep 10 23:06:23.980561 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.980495    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 23:06:23.997770 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:23.997745    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"]
Sep 10 23:06:24.049535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.049501    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slj7m\" (UniqueName: \"kubernetes.io/projected/c976b377-6cc2-47cd-a066-971f985b1393-kube-api-access-slj7m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:24.049679 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.049620    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c976b377-6cc2-47cd-a066-971f985b1393-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:24.150726 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.150685    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c976b377-6cc2-47cd-a066-971f985b1393-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:24.150726 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.150725    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slj7m\" (UniqueName: \"kubernetes.io/projected/c976b377-6cc2-47cd-a066-971f985b1393-kube-api-access-slj7m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:24.151082 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.151060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c976b377-6cc2-47cd-a066-971f985b1393-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:24.163351 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.163324    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slj7m\" (UniqueName: \"kubernetes.io/projected/c976b377-6cc2-47cd-a066-971f985b1393-kube-api-access-slj7m\") pod \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:24.287528 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.287425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:24.499452 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.499420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:24.567054 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:24.567028    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"]
Sep 10 23:06:24.571442 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:06:24.571409    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc976b377_6cc2_47cd_a066_971f985b1393.slice/crio-f4916a1490737132986c0d5c7c660f230e2a268a3a92b7f6393c99033bf44505 WatchSource:0}: Error finding container f4916a1490737132986c0d5c7c660f230e2a268a3a92b7f6393c99033bf44505: Status 404 returned error can't find the container with id f4916a1490737132986c0d5c7c660f230e2a268a3a92b7f6393c99033bf44505
Sep 10 23:06:25.104704 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:25.104670    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h" event={"ID":"c976b377-6cc2-47cd-a066-971f985b1393","Type":"ContainerStarted","Data":"f4916a1490737132986c0d5c7c660f230e2a268a3a92b7f6393c99033bf44505"}
Sep 10 23:06:26.158976 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:26.158903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:26.403750 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:26.403711    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:26.527026 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:26.526915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:28.565518 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:28.565482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:28.672607 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:28.672571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:29.126813 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:29.126760    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h" event={"ID":"c976b377-6cc2-47cd-a066-971f985b1393","Type":"ContainerStarted","Data":"34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970"}
Sep 10 23:06:29.126992 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:29.126914    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:29.196245 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:29.196187    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h" podStartSLOduration=2.326741577 podStartE2EDuration="6.196170025s" totalImagesPullingTime="3.869428448s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:29.19385515 +0000 UTC m=+505.629909924" watchObservedRunningTime="2026-09-10 23:06:29.196170025 +0000 UTC m=+505.632224797"
Sep 10 23:06:29.354913 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:29.354876    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:29.931074 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:29.931046    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:30.417671 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:30.417637    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:30.501762 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:30.501727    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:31.019726 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.019690    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"]
Sep 10 23:06:31.023097 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.023074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"
Sep 10 23:06:31.025642 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.025620    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 23:06:31.025735 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.025620    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-7csxr\""
Sep 10 23:06:31.036766 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.036742    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"]
Sep 10 23:06:31.114998 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.114964    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7fp9\" (UniqueName: \"kubernetes.io/projected/17823acd-5e64-4b9e-bbb8-0f028352b7f7-kube-api-access-h7fp9\") pod \"dns-operator-controller-manager-7b84cd7c9c-tvcjg\" (UID: \"17823acd-5e64-4b9e-bbb8-0f028352b7f7\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"
Sep 10 23:06:31.216136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.216098    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7fp9\" (UniqueName: \"kubernetes.io/projected/17823acd-5e64-4b9e-bbb8-0f028352b7f7-kube-api-access-h7fp9\") pod \"dns-operator-controller-manager-7b84cd7c9c-tvcjg\" (UID: \"17823acd-5e64-4b9e-bbb8-0f028352b7f7\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"
Sep 10 23:06:31.230523 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.230492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7fp9\" (UniqueName: \"kubernetes.io/projected/17823acd-5e64-4b9e-bbb8-0f028352b7f7-kube-api-access-h7fp9\") pod \"dns-operator-controller-manager-7b84cd7c9c-tvcjg\" (UID: \"17823acd-5e64-4b9e-bbb8-0f028352b7f7\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"
Sep 10 23:06:31.332433 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.332348    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"
Sep 10 23:06:31.598075 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.597984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:31.645275 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.645240    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"]
Sep 10 23:06:31.653024 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:06:31.652995    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17823acd_5e64_4b9e_bbb8_0f028352b7f7.slice/crio-a632865c7a594db90a9f10908b112a36925b697b36c74793f2819aebe2f0929b WatchSource:0}: Error finding container a632865c7a594db90a9f10908b112a36925b697b36c74793f2819aebe2f0929b: Status 404 returned error can't find the container with id a632865c7a594db90a9f10908b112a36925b697b36c74793f2819aebe2f0929b
Sep 10 23:06:31.908162 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:31.908135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:32.145738 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:32.145695    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg" event={"ID":"17823acd-5e64-4b9e-bbb8-0f028352b7f7","Type":"ContainerStarted","Data":"a632865c7a594db90a9f10908b112a36925b697b36c74793f2819aebe2f0929b"}
Sep 10 23:06:34.156289 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:34.156252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg" event={"ID":"17823acd-5e64-4b9e-bbb8-0f028352b7f7","Type":"ContainerStarted","Data":"2dda86ee38963128582e5c776ddb078b3be5f180726d7e577ecf408589ae9cef"}
Sep 10 23:06:34.156717 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:34.156383    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"
Sep 10 23:06:34.178294 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:34.178240    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg" podStartSLOduration=2.659372304 podStartE2EDuration="4.178224201s" totalImagesPullingTime="1.518851897s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:34.176810039 +0000 UTC m=+510.612864805" watchObservedRunningTime="2026-09-10 23:06:34.178224201 +0000 UTC m=+510.614278974"
Sep 10 23:06:34.359415 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:34.359374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:34.449842 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:34.449720    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:34.756831 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:34.756734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:34.856388 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:34.856337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:35.112605 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:35.112496    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:35.218948 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:35.218904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:40.133674 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:40.133642    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:41.415602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:41.415567    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:41.523402 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:41.523346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:42.455410 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.455373    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"]
Sep 10 23:06:42.461587 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.461567    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:42.485208 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.485177    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"]
Sep 10 23:06:42.586963 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.586930    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"]
Sep 10 23:06:42.587191 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:42.587172    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[extensions-socket-volume kube-api-access-2mmnk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk" podUID="f0b97ea0-8498-4b5e-aed3-fd1e35ff7198"
Sep 10 23:06:42.610768 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.610727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" (UID: \"f0b97ea0-8498-4b5e-aed3-fd1e35ff7198\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:42.610943 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.610773    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mmnk\" (UniqueName: \"kubernetes.io/projected/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-kube-api-access-2mmnk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" (UID: \"f0b97ea0-8498-4b5e-aed3-fd1e35ff7198\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:42.616271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.616246    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"]
Sep 10 23:06:42.638747 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.638712    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"]
Sep 10 23:06:42.638964 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.638941    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h" podUID="c976b377-6cc2-47cd-a066-971f985b1393" containerName="manager" containerID="cri-o://34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970" gracePeriod=2
Sep 10 23:06:42.655241 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.655210    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"]
Sep 10 23:06:42.675358 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.675326    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"]
Sep 10 23:06:42.675735 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.675694    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c976b377-6cc2-47cd-a066-971f985b1393" containerName="manager"
Sep 10 23:06:42.675735 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.675715    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c976b377-6cc2-47cd-a066-971f985b1393" containerName="manager"
Sep 10 23:06:42.675924 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.675836    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c976b377-6cc2-47cd-a066-971f985b1393" containerName="manager"
Sep 10 23:06:42.679296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.679271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:42.681394 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:42.681360    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:42.706745 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.706663    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"]
Sep 10 23:06:42.711646 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.711617    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" (UID: \"f0b97ea0-8498-4b5e-aed3-fd1e35ff7198\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:42.711816 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.711656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mmnk\" (UniqueName: \"kubernetes.io/projected/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-kube-api-access-2mmnk\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" (UID: \"f0b97ea0-8498-4b5e-aed3-fd1e35ff7198\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:42.712135 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.712111    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" (UID: \"f0b97ea0-8498-4b5e-aed3-fd1e35ff7198\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:42.720884 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:42.720855    2577 projected.go:196] Error preparing data for projected volume kube-api-access-2mmnk for pod kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk: failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-v5jmk" not found
Sep 10 23:06:42.721019 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:42.720919    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-kube-api-access-2mmnk podName:f0b97ea0-8498-4b5e-aed3-fd1e35ff7198 nodeName:}" failed. No retries permitted until 2026-09-10 23:06:43.220903422 +0000 UTC m=+519.656958173 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2mmnk" (UniqueName: "kubernetes.io/projected/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-kube-api-access-2mmnk") pod "kuadrant-operator-controller-manager-5769f99d9f-v5jmk" (UID: "f0b97ea0-8498-4b5e-aed3-fd1e35ff7198") : failed to fetch token: pod "kuadrant-operator-controller-manager-5769f99d9f-v5jmk" not found
Sep 10 23:06:42.726195 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.726170    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"]
Sep 10 23:06:42.729874 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.729850    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"
Sep 10 23:06:42.732409 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.732388    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kfn4h\""
Sep 10 23:06:42.741827 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:42.741773    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:42.746288 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.746265    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"]
Sep 10 23:06:42.812197 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.812163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed179f7-f545-45a0-b320-cf025967422e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8nn5z\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:42.812363 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.812204    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhjvf\" (UniqueName: \"kubernetes.io/projected/0ed179f7-f545-45a0-b320-cf025967422e-kube-api-access-mhjvf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8nn5z\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:42.812420 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.812348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7t6w\" (UniqueName: \"kubernetes.io/projected/17aacf39-4d0f-4402-99b9-63d60bb6422f-kube-api-access-s7t6w\") pod \"limitador-operator-controller-manager-74495c69f8-m76rp\" (UID: \"17aacf39-4d0f-4402-99b9-63d60bb6422f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"
Sep 10 23:06:42.883175 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.883149    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:42.885731 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:42.885708    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:42.913803 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.913749    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed179f7-f545-45a0-b320-cf025967422e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8nn5z\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:42.913962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.913900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhjvf\" (UniqueName: \"kubernetes.io/projected/0ed179f7-f545-45a0-b320-cf025967422e-kube-api-access-mhjvf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8nn5z\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:42.914029 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.913998    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7t6w\" (UniqueName: \"kubernetes.io/projected/17aacf39-4d0f-4402-99b9-63d60bb6422f-kube-api-access-s7t6w\") pod \"limitador-operator-controller-manager-74495c69f8-m76rp\" (UID: \"17aacf39-4d0f-4402-99b9-63d60bb6422f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"
Sep 10 23:06:42.914183 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.914162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed179f7-f545-45a0-b320-cf025967422e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8nn5z\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:42.945373 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.945350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhjvf\" (UniqueName: \"kubernetes.io/projected/0ed179f7-f545-45a0-b320-cf025967422e-kube-api-access-mhjvf\") pod \"kuadrant-operator-controller-manager-76f8f686bf-8nn5z\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:42.947422 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:42.947403    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7t6w\" (UniqueName: \"kubernetes.io/projected/17aacf39-4d0f-4402-99b9-63d60bb6422f-kube-api-access-s7t6w\") pod \"limitador-operator-controller-manager-74495c69f8-m76rp\" (UID: \"17aacf39-4d0f-4402-99b9-63d60bb6422f\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"
Sep 10 23:06:43.015154 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.015071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slj7m\" (UniqueName: \"kubernetes.io/projected/c976b377-6cc2-47cd-a066-971f985b1393-kube-api-access-slj7m\") pod \"c976b377-6cc2-47cd-a066-971f985b1393\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") "
Sep 10 23:06:43.015154 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.015109    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c976b377-6cc2-47cd-a066-971f985b1393-extensions-socket-volume\") pod \"c976b377-6cc2-47cd-a066-971f985b1393\" (UID: \"c976b377-6cc2-47cd-a066-971f985b1393\") "
Sep 10 23:06:43.015615 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.015589    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c976b377-6cc2-47cd-a066-971f985b1393-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "c976b377-6cc2-47cd-a066-971f985b1393" (UID: "c976b377-6cc2-47cd-a066-971f985b1393"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:06:43.017310 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.017287    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c976b377-6cc2-47cd-a066-971f985b1393-kube-api-access-slj7m" (OuterVolumeSpecName: "kube-api-access-slj7m") pod "c976b377-6cc2-47cd-a066-971f985b1393" (UID: "c976b377-6cc2-47cd-a066-971f985b1393"). InnerVolumeSpecName "kube-api-access-slj7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:06:43.022439 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.022422    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:43.040393 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.040367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"
Sep 10 23:06:43.116471 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.116430    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slj7m\" (UniqueName: \"kubernetes.io/projected/c976b377-6cc2-47cd-a066-971f985b1393-kube-api-access-slj7m\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:06:43.116471 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.116469    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c976b377-6cc2-47cd-a066-971f985b1393-extensions-socket-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:06:43.195664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.195626    2577 generic.go:356] "Generic (PLEG): container finished" podID="c976b377-6cc2-47cd-a066-971f985b1393" containerID="34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970" exitCode=0
Sep 10 23:06:43.195859 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.195672    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:43.195859 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.195726    2577 scope.go:117] "RemoveContainer" containerID="34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970"
Sep 10 23:06:43.195991 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.195971    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:43.197944 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.197913    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:43.200044 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.200020    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="f0b97ea0-8498-4b5e-aed3-fd1e35ff7198" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:43.202425 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.202394    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:43.204388 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.204367    2577 scope.go:117] "RemoveContainer" containerID="34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970"
Sep 10 23:06:43.204703 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.204685    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970\": container with ID starting with 34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970 not found: ID does not exist" containerID="34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970"
Sep 10 23:06:43.204769 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.204712    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970"} err="failed to get container status \"34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970\": rpc error: code = NotFound desc = could not find container \"34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970\": container with ID starting with 34523ef0c11d6a524361305cdf2e4704e635ab51734ffd86730661bd6c790970 not found: ID does not exist"
Sep 10 23:06:43.206497 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.206479    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:43.208603 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.208581    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="f0b97ea0-8498-4b5e-aed3-fd1e35ff7198" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:43.211852 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.211831    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:43.214379 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.214358    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="f0b97ea0-8498-4b5e-aed3-fd1e35ff7198" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:43.216141 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:43.216118    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:43.318078 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.317996    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-extensions-socket-volume\") pod \"f0b97ea0-8498-4b5e-aed3-fd1e35ff7198\" (UID: \"f0b97ea0-8498-4b5e-aed3-fd1e35ff7198\") "
Sep 10 23:06:43.318295 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.318277    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mmnk\" (UniqueName: \"kubernetes.io/projected/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-kube-api-access-2mmnk\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:06:43.318337 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.318278    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "f0b97ea0-8498-4b5e-aed3-fd1e35ff7198" (UID: "f0b97ea0-8498-4b5e-aed3-fd1e35ff7198"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:06:43.326824 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:06:43.326772    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed179f7_f545_45a0_b320_cf025967422e.slice/crio-4fc48947a71aa11e029c83c58e7183b4fdefa9872d559e3422c1624736db373d WatchSource:0}: Error finding container 4fc48947a71aa11e029c83c58e7183b4fdefa9872d559e3422c1624736db373d: Status 404 returned error can't find the container with id 4fc48947a71aa11e029c83c58e7183b4fdefa9872d559e3422c1624736db373d
Sep 10 23:06:43.332248 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.332223    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"]
Sep 10 23:06:43.342631 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.342607    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"]
Sep 10 23:06:43.344357 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:06:43.344332    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17aacf39_4d0f_4402_99b9_63d60bb6422f.slice/crio-272468efb49af95680f06ae0af59aea779bd6a57fa7acce6c7c3951ab28ce81f WatchSource:0}: Error finding container 272468efb49af95680f06ae0af59aea779bd6a57fa7acce6c7c3951ab28ce81f: Status 404 returned error can't find the container with id 272468efb49af95680f06ae0af59aea779bd6a57fa7acce6c7c3951ab28ce81f
Sep 10 23:06:43.419638 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:43.419599    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198-extensions-socket-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:06:44.201858 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.201814    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp" event={"ID":"17aacf39-4d0f-4402-99b9-63d60bb6422f","Type":"ContainerStarted","Data":"272468efb49af95680f06ae0af59aea779bd6a57fa7acce6c7c3951ab28ce81f"}
Sep 10 23:06:44.205096 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.205063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:44.205096 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.205084    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z" event={"ID":"0ed179f7-f545-45a0-b320-cf025967422e","Type":"ContainerStarted","Data":"7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f"}
Sep 10 23:06:44.205399 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.205128    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z" event={"ID":"0ed179f7-f545-45a0-b320-cf025967422e","Type":"ContainerStarted","Data":"4fc48947a71aa11e029c83c58e7183b4fdefa9872d559e3422c1624736db373d"}
Sep 10 23:06:44.205510 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.205487    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:44.207445 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:44.207418    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="f0b97ea0-8498-4b5e-aed3-fd1e35ff7198" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:44.209771 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:44.209742    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:44.214371 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.214345    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c976b377-6cc2-47cd-a066-971f985b1393" path="/var/lib/kubelet/pods/c976b377-6cc2-47cd-a066-971f985b1393/volumes"
Sep 10 23:06:44.214704 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.214690    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0b97ea0-8498-4b5e-aed3-fd1e35ff7198" path="/var/lib/kubelet/pods/f0b97ea0-8498-4b5e-aed3-fd1e35ff7198/volumes"
Sep 10 23:06:44.215418 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:44.215397    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-5769f99d9f-v5jmk\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="f0b97ea0-8498-4b5e-aed3-fd1e35ff7198" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-v5jmk"
Sep 10 23:06:44.217306 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:44.217280    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-grv5h\" is forbidden: User \"system:node:ip-10-0-138-150.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-138-150.ec2.internal' and this object" podUID="c976b377-6cc2-47cd-a066-971f985b1393" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-grv5h"
Sep 10 23:06:44.251468 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:44.251351    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z" podStartSLOduration=2.251327529 podStartE2EDuration="2.251327529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:44.250971128 +0000 UTC m=+520.687025901" watchObservedRunningTime="2026-09-10 23:06:44.251327529 +0000 UTC m=+520.687382302"
Sep 10 23:06:45.163111 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:45.163082    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-tvcjg"
Sep 10 23:06:45.211067 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:45.211025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp" event={"ID":"17aacf39-4d0f-4402-99b9-63d60bb6422f","Type":"ContainerStarted","Data":"4db010d758456a19c8d19cd8685e33f5c21fb7587398bf699df6848ec979dbf1"}
Sep 10 23:06:45.269937 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:45.269885    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp" podStartSLOduration=1.84622221 podStartE2EDuration="3.269865978s" totalImagesPullingTime="1.423643768s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:06:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:06:45.266844303 +0000 UTC m=+521.702899073" watchObservedRunningTime="2026-09-10 23:06:45.269865978 +0000 UTC m=+521.705920751"
Sep 10 23:06:46.190896 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:46.190852    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:46.215206 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:46.215173    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"
Sep 10 23:06:46.291614 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:46.291582    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:48.630832 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:48.630800    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:48.738229 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:48.738197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:55.213931 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:55.213895    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:55.588487 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:55.588399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:55.681758 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:55.681717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:55.931359 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:55.931327    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:56.013389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:56.013353    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:56.144592 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:56.144506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:56.248930 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:56.248848    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:06:57.222402 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:57.222374    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-m76rp"
Sep 10 23:06:57.888721 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:57.888685    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"]
Sep 10 23:06:57.889168 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:57.888945    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z" podUID="0ed179f7-f545-45a0-b320-cf025967422e" containerName="manager" containerID="cri-o://7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f" gracePeriod=10
Sep 10 23:06:58.136958 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.136928    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:58.244374 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.244335    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mhjvf\" (UniqueName: \"kubernetes.io/projected/0ed179f7-f545-45a0-b320-cf025967422e-kube-api-access-mhjvf\") pod \"0ed179f7-f545-45a0-b320-cf025967422e\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") "
Sep 10 23:06:58.244576 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.244422    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed179f7-f545-45a0-b320-cf025967422e-extensions-socket-volume\") pod \"0ed179f7-f545-45a0-b320-cf025967422e\" (UID: \"0ed179f7-f545-45a0-b320-cf025967422e\") "
Sep 10 23:06:58.244833 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.244776    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ed179f7-f545-45a0-b320-cf025967422e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "0ed179f7-f545-45a0-b320-cf025967422e" (UID: "0ed179f7-f545-45a0-b320-cf025967422e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:06:58.246507 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.246481    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ed179f7-f545-45a0-b320-cf025967422e-kube-api-access-mhjvf" (OuterVolumeSpecName: "kube-api-access-mhjvf") pod "0ed179f7-f545-45a0-b320-cf025967422e" (UID: "0ed179f7-f545-45a0-b320-cf025967422e"). InnerVolumeSpecName "kube-api-access-mhjvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:06:58.274911 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.274876    2577 generic.go:356] "Generic (PLEG): container finished" podID="0ed179f7-f545-45a0-b320-cf025967422e" containerID="7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f" exitCode=0
Sep 10 23:06:58.275081 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.274922    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"
Sep 10 23:06:58.275081 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.274947    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z" event={"ID":"0ed179f7-f545-45a0-b320-cf025967422e","Type":"ContainerDied","Data":"7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f"}
Sep 10 23:06:58.275081 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.274999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z" event={"ID":"0ed179f7-f545-45a0-b320-cf025967422e","Type":"ContainerDied","Data":"4fc48947a71aa11e029c83c58e7183b4fdefa9872d559e3422c1624736db373d"}
Sep 10 23:06:58.275081 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.275028    2577 scope.go:117] "RemoveContainer" containerID="7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f"
Sep 10 23:06:58.283927 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.283909    2577 scope.go:117] "RemoveContainer" containerID="7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f"
Sep 10 23:06:58.284247 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:06:58.284229    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f\": container with ID starting with 7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f not found: ID does not exist" containerID="7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f"
Sep 10 23:06:58.284327 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.284258    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f"} err="failed to get container status \"7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f\": rpc error: code = NotFound desc = could not find container \"7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f\": container with ID starting with 7fa376a7456e72055b04e371bce49d94ce11d95c11fe95186b2fa10ca5fb024f not found: ID does not exist"
Sep 10 23:06:58.307677 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.307644    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"]
Sep 10 23:06:58.322349 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.322318    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-8nn5z"]
Sep 10 23:06:58.345815 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.345773    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhjvf\" (UniqueName: \"kubernetes.io/projected/0ed179f7-f545-45a0-b320-cf025967422e-kube-api-access-mhjvf\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:06:58.345815 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:06:58.345812    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/0ed179f7-f545-45a0-b320-cf025967422e-extensions-socket-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:07:00.214086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:00.214051    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed179f7-f545-45a0-b320-cf025967422e" path="/var/lib/kubelet/pods/0ed179f7-f545-45a0-b320-cf025967422e/volumes"
Sep 10 23:07:03.150611 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:03.150573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:03.243972 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:03.243910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:04.256460 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:04.256422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:04.373377 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:04.373338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:07.850856 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:07.850820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:07.956502 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:07.956461    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:08.071815 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:08.071761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:10.399716 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:10.399681    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:10.501011 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:10.500975    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:14.395218 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:14.395163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:14.751492 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:14.751443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:17.133387 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.133350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:17.164898 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.164860    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nb26k"]
Sep 10 23:07:17.165185 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.165171    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed179f7-f545-45a0-b320-cf025967422e" containerName="manager"
Sep 10 23:07:17.165232 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.165187    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed179f7-f545-45a0-b320-cf025967422e" containerName="manager"
Sep 10 23:07:17.165273 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.165247    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed179f7-f545-45a0-b320-cf025967422e" containerName="manager"
Sep 10 23:07:17.167960 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.167941    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.171560 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.171535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 23:07:17.171696 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.171678    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-vpt4b\""
Sep 10 23:07:17.179753 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.179721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-257pk\" (UniqueName: \"kubernetes.io/projected/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-kube-api-access-257pk\") pod \"limitador-limitador-8588749977-nb26k\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") " pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.179908 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.179770    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-config-file\") pod \"limitador-limitador-8588749977-nb26k\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") " pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.181464 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.181437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nb26k"]
Sep 10 23:07:17.246533 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.246501    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nb26k"]
Sep 10 23:07:17.266829 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.266753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:17.281112 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.281069    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-257pk\" (UniqueName: \"kubernetes.io/projected/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-kube-api-access-257pk\") pod \"limitador-limitador-8588749977-nb26k\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") " pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.281276 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.281120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-config-file\") pod \"limitador-limitador-8588749977-nb26k\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") " pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.281765 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.281746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-config-file\") pod \"limitador-limitador-8588749977-nb26k\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") " pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.293930 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.293902    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-257pk\" (UniqueName: \"kubernetes.io/projected/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-kube-api-access-257pk\") pod \"limitador-limitador-8588749977-nb26k\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") " pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.332828 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.332762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:17.478095 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.478059    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:17.755327 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:17.755161    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nb26k"]
Sep 10 23:07:17.758177 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:07:17.758148    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3270a3cf_3c24_40e6_80ed_e6d64d00d9c2.slice/crio-d931a5084f09be53d51174ed1d6ab314a7c7b4e913677bcef2fd7d7323981930 WatchSource:0}: Error finding container d931a5084f09be53d51174ed1d6ab314a7c7b4e913677bcef2fd7d7323981930: Status 404 returned error can't find the container with id d931a5084f09be53d51174ed1d6ab314a7c7b4e913677bcef2fd7d7323981930
Sep 10 23:07:18.183122 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.183089    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-slqg7"]
Sep 10 23:07:18.185915 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.185893    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7"
Sep 10 23:07:18.188202 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.188181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzgsr\""
Sep 10 23:07:18.199098 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.199074    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-slqg7"]
Sep 10 23:07:18.224266 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.224233    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:18.289154 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.289115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4gpc6\" (UniqueName: \"kubernetes.io/projected/98a55603-e5ea-4970-b3ff-7fdd0460e838-kube-api-access-4gpc6\") pod \"authorino-7bcfd8f9cd-slqg7\" (UID: \"98a55603-e5ea-4970-b3ff-7fdd0460e838\") " pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7"
Sep 10 23:07:18.297046 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.297021    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:18.367897 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.367860    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nb26k" event={"ID":"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2","Type":"ContainerStarted","Data":"d931a5084f09be53d51174ed1d6ab314a7c7b4e913677bcef2fd7d7323981930"}
Sep 10 23:07:18.390465 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.390427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4gpc6\" (UniqueName: \"kubernetes.io/projected/98a55603-e5ea-4970-b3ff-7fdd0460e838-kube-api-access-4gpc6\") pod \"authorino-7bcfd8f9cd-slqg7\" (UID: \"98a55603-e5ea-4970-b3ff-7fdd0460e838\") " pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7"
Sep 10 23:07:18.412973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.412945    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4gpc6\" (UniqueName: \"kubernetes.io/projected/98a55603-e5ea-4970-b3ff-7fdd0460e838-kube-api-access-4gpc6\") pod \"authorino-7bcfd8f9cd-slqg7\" (UID: \"98a55603-e5ea-4970-b3ff-7fdd0460e838\") " pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7"
Sep 10 23:07:18.495001 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.494908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7"
Sep 10 23:07:18.795765 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:18.795738    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-slqg7"]
Sep 10 23:07:18.799614 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:07:18.799585    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod98a55603_e5ea_4970_b3ff_7fdd0460e838.slice/crio-ee4ed9506702d95f1841b5e9112e5696e5a564986b3a9b7012b44ad10bb4a627 WatchSource:0}: Error finding container ee4ed9506702d95f1841b5e9112e5696e5a564986b3a9b7012b44ad10bb4a627: Status 404 returned error can't find the container with id ee4ed9506702d95f1841b5e9112e5696e5a564986b3a9b7012b44ad10bb4a627
Sep 10 23:07:19.373981 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:19.373943    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nb26k" event={"ID":"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2","Type":"ContainerStarted","Data":"3b7cca724cc591d7589f2ace0eb073c105a988d2806d43db86425cc71c8230a7"}
Sep 10 23:07:19.374409 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:19.374024    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:19.375144 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:19.375124    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7" event={"ID":"98a55603-e5ea-4970-b3ff-7fdd0460e838","Type":"ContainerStarted","Data":"ee4ed9506702d95f1841b5e9112e5696e5a564986b3a9b7012b44ad10bb4a627"}
Sep 10 23:07:19.394667 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:19.394611    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-nb26k" podStartSLOduration=1.390225697 podStartE2EDuration="2.394593371s" totalImagesPullingTime="1.004367674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:07:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:07:19.392431786 +0000 UTC m=+555.828486560" watchObservedRunningTime="2026-09-10 23:07:19.394593371 +0000 UTC m=+555.830648144"
Sep 10 23:07:20.885676 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:20.885583    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-slqg7"]
Sep 10 23:07:21.388208 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:21.388173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7" event={"ID":"98a55603-e5ea-4970-b3ff-7fdd0460e838","Type":"ContainerStarted","Data":"47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a"}
Sep 10 23:07:21.413001 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:21.412943    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7" podStartSLOduration=1.750132517 podStartE2EDuration="3.412926787s" totalImagesPullingTime="1.66279427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:07:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:07:21.411719492 +0000 UTC m=+557.847774281" watchObservedRunningTime="2026-09-10 23:07:21.412926787 +0000 UTC m=+557.848981559"
Sep 10 23:07:22.393292 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:22.393228    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7" podUID="98a55603-e5ea-4970-b3ff-7fdd0460e838" containerName="authorino" containerID="cri-o://47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a" gracePeriod=30
Sep 10 23:07:22.634314 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:22.634288    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7"
Sep 10 23:07:22.724028 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:22.723993    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4gpc6\" (UniqueName: \"kubernetes.io/projected/98a55603-e5ea-4970-b3ff-7fdd0460e838-kube-api-access-4gpc6\") pod \"98a55603-e5ea-4970-b3ff-7fdd0460e838\" (UID: \"98a55603-e5ea-4970-b3ff-7fdd0460e838\") "
Sep 10 23:07:22.726200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:22.726162    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98a55603-e5ea-4970-b3ff-7fdd0460e838-kube-api-access-4gpc6" (OuterVolumeSpecName: "kube-api-access-4gpc6") pod "98a55603-e5ea-4970-b3ff-7fdd0460e838" (UID: "98a55603-e5ea-4970-b3ff-7fdd0460e838"). InnerVolumeSpecName "kube-api-access-4gpc6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:07:22.825449 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:22.825413    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4gpc6\" (UniqueName: \"kubernetes.io/projected/98a55603-e5ea-4970-b3ff-7fdd0460e838-kube-api-access-4gpc6\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:07:23.399298 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.399259    2577 generic.go:356] "Generic (PLEG): container finished" podID="98a55603-e5ea-4970-b3ff-7fdd0460e838" containerID="47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a" exitCode=0
Sep 10 23:07:23.399298 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.399302    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7"
Sep 10 23:07:23.399769 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.399350    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7" event={"ID":"98a55603-e5ea-4970-b3ff-7fdd0460e838","Type":"ContainerDied","Data":"47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a"}
Sep 10 23:07:23.399769 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.399397    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-slqg7" event={"ID":"98a55603-e5ea-4970-b3ff-7fdd0460e838","Type":"ContainerDied","Data":"ee4ed9506702d95f1841b5e9112e5696e5a564986b3a9b7012b44ad10bb4a627"}
Sep 10 23:07:23.399769 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.399418    2577 scope.go:117] "RemoveContainer" containerID="47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a"
Sep 10 23:07:23.410016 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.409994    2577 scope.go:117] "RemoveContainer" containerID="47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a"
Sep 10 23:07:23.410316 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:07:23.410297    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a\": container with ID starting with 47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a not found: ID does not exist" containerID="47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a"
Sep 10 23:07:23.410364 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.410326    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a"} err="failed to get container status \"47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a\": rpc error: code = NotFound desc = could not find container \"47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a\": container with ID starting with 47bc4adb2e918c8a6f26d4c221562ab93adf22725280aa4d0253b8b3d0f1898a not found: ID does not exist"
Sep 10 23:07:23.424097 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.424066    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-slqg7"]
Sep 10 23:07:23.430562 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:23.430534    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-slqg7"]
Sep 10 23:07:24.215463 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:24.215432    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98a55603-e5ea-4970-b3ff-7fdd0460e838" path="/var/lib/kubelet/pods/98a55603-e5ea-4970-b3ff-7fdd0460e838/volumes"
Sep 10 23:07:28.964100 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:28.964062    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:30.384471 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:30.384444    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:30.891098 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:30.891056    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nb26k"]
Sep 10 23:07:30.891283 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:30.891261    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-nb26k" podUID="3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" containerName="limitador" containerID="cri-o://3b7cca724cc591d7589f2ace0eb073c105a988d2806d43db86425cc71c8230a7" gracePeriod=30
Sep 10 23:07:31.437501 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.437464    2577 generic.go:356] "Generic (PLEG): container finished" podID="3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" containerID="3b7cca724cc591d7589f2ace0eb073c105a988d2806d43db86425cc71c8230a7" exitCode=0
Sep 10 23:07:31.437817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.437576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nb26k" event={"ID":"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2","Type":"ContainerDied","Data":"3b7cca724cc591d7589f2ace0eb073c105a988d2806d43db86425cc71c8230a7"}
Sep 10 23:07:31.437817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.437637    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-nb26k" event={"ID":"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2","Type":"ContainerDied","Data":"d931a5084f09be53d51174ed1d6ab314a7c7b4e913677bcef2fd7d7323981930"}
Sep 10 23:07:31.437817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.437660    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d931a5084f09be53d51174ed1d6ab314a7c7b4e913677bcef2fd7d7323981930"
Sep 10 23:07:31.439753 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.439739    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:31.498252 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.498168    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-config-file\") pod \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") "
Sep 10 23:07:31.498252 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.498232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-257pk\" (UniqueName: \"kubernetes.io/projected/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-kube-api-access-257pk\") pod \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\" (UID: \"3270a3cf-3c24-40e6-80ed-e6d64d00d9c2\") "
Sep 10 23:07:31.498550 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.498527    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-config-file" (OuterVolumeSpecName: "config-file") pod "3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" (UID: "3270a3cf-3c24-40e6-80ed-e6d64d00d9c2"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:07:31.500360 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.500328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-kube-api-access-257pk" (OuterVolumeSpecName: "kube-api-access-257pk") pod "3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" (UID: "3270a3cf-3c24-40e6-80ed-e6d64d00d9c2"). InnerVolumeSpecName "kube-api-access-257pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:07:31.599697 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.599658    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-config-file\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:07:31.599697 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:31.599692    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-257pk\" (UniqueName: \"kubernetes.io/projected/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2-kube-api-access-257pk\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:07:32.442163 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:32.442127    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-nb26k"
Sep 10 23:07:32.477708 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:32.477619    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nb26k"]
Sep 10 23:07:32.487141 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:32.487106    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-nb26k"]
Sep 10 23:07:33.241819 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.241762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:33.336874 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.336823    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:33.685949 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.685915    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"]
Sep 10 23:07:33.686391 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.686377    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" containerName="limitador"
Sep 10 23:07:33.686457 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.686396    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" containerName="limitador"
Sep 10 23:07:33.686457 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.686410    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98a55603-e5ea-4970-b3ff-7fdd0460e838" containerName="authorino"
Sep 10 23:07:33.686457 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.686418    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="98a55603-e5ea-4970-b3ff-7fdd0460e838" containerName="authorino"
Sep 10 23:07:33.686600 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.686508    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" containerName="limitador"
Sep 10 23:07:33.686600 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.686519    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="98a55603-e5ea-4970-b3ff-7fdd0460e838" containerName="authorino"
Sep 10 23:07:33.689409 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.689388    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:33.691912 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.691889    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-xrp6l\""
Sep 10 23:07:33.692143 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.692123    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 23:07:33.692716 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.692694    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 23:07:33.692851 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.692697    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 23:07:33.704504 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.704480    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"]
Sep 10 23:07:33.714413 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.714388    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcnkx\" (UniqueName: \"kubernetes.io/projected/a351ffe3-6f4e-43bd-9563-78fde50c0490-kube-api-access-pcnkx\") pod \"postgres-69fb4594cf-c7xml\" (UID: \"a351ffe3-6f4e-43bd-9563-78fde50c0490\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:33.714509 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.714467    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a351ffe3-6f4e-43bd-9563-78fde50c0490-data\") pod \"postgres-69fb4594cf-c7xml\" (UID: \"a351ffe3-6f4e-43bd-9563-78fde50c0490\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:33.731792 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.731763    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:33.815798 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.815743    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcnkx\" (UniqueName: \"kubernetes.io/projected/a351ffe3-6f4e-43bd-9563-78fde50c0490-kube-api-access-pcnkx\") pod \"postgres-69fb4594cf-c7xml\" (UID: \"a351ffe3-6f4e-43bd-9563-78fde50c0490\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:33.815952 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.815830    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a351ffe3-6f4e-43bd-9563-78fde50c0490-data\") pod \"postgres-69fb4594cf-c7xml\" (UID: \"a351ffe3-6f4e-43bd-9563-78fde50c0490\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:33.816158 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.816143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/a351ffe3-6f4e-43bd-9563-78fde50c0490-data\") pod \"postgres-69fb4594cf-c7xml\" (UID: \"a351ffe3-6f4e-43bd-9563-78fde50c0490\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:33.826506 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.826480    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcnkx\" (UniqueName: \"kubernetes.io/projected/a351ffe3-6f4e-43bd-9563-78fde50c0490-kube-api-access-pcnkx\") pod \"postgres-69fb4594cf-c7xml\" (UID: \"a351ffe3-6f4e-43bd-9563-78fde50c0490\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:33.999837 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:33.999727    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:34.214380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:34.214348    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3270a3cf-3c24-40e6-80ed-e6d64d00d9c2" path="/var/lib/kubelet/pods/3270a3cf-3c24-40e6-80ed-e6d64d00d9c2/volumes"
Sep 10 23:07:34.284325 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:34.284247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"]
Sep 10 23:07:34.287153 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:07:34.287121    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda351ffe3_6f4e_43bd_9563_78fde50c0490.slice/crio-e03243c52f1f0cff616f78d327d256b7487b759dec4868f4c8a333c3ba27263e WatchSource:0}: Error finding container e03243c52f1f0cff616f78d327d256b7487b759dec4868f4c8a333c3ba27263e: Status 404 returned error can't find the container with id e03243c52f1f0cff616f78d327d256b7487b759dec4868f4c8a333c3ba27263e
Sep 10 23:07:34.452572 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:34.452540    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml" event={"ID":"a351ffe3-6f4e-43bd-9563-78fde50c0490","Type":"ContainerStarted","Data":"e03243c52f1f0cff616f78d327d256b7487b759dec4868f4c8a333c3ba27263e"}
Sep 10 23:07:39.478814 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:39.478759    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml" event={"ID":"a351ffe3-6f4e-43bd-9563-78fde50c0490","Type":"ContainerStarted","Data":"88586998d91d309545aa14458cb2a048fa92881f885fd53ff00c2a8106fd483e"}
Sep 10 23:07:39.479274 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:39.478897    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:39.499739 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:39.499693    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml" podStartSLOduration=1.690276392 podStartE2EDuration="6.499679374s" totalImagesPullingTime="4.809402982s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:07:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:07:39.499287017 +0000 UTC m=+575.935341790" watchObservedRunningTime="2026-09-10 23:07:39.499679374 +0000 UTC m=+575.935734211"
Sep 10 23:07:45.510443 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:45.510412    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-c7xml"
Sep 10 23:07:45.851495 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:45.851413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:46.098479 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.098441    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-54vqt"]
Sep 10 23:07:46.103969 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.103918    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54vqt"
Sep 10 23:07:46.107226 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.107205    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-zzgsr\""
Sep 10 23:07:46.118853 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.118825    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-54vqt"]
Sep 10 23:07:46.226194 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.226164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/8e32da86-0b51-4502-b85d-73d1671c92f0-kube-api-access-kfzc5\") pod \"authorino-557f876f69-54vqt\" (UID: \"8e32da86-0b51-4502-b85d-73d1671c92f0\") " pod="kuadrant-system/authorino-557f876f69-54vqt"
Sep 10 23:07:46.326694 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.326660    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/8e32da86-0b51-4502-b85d-73d1671c92f0-kube-api-access-kfzc5\") pod \"authorino-557f876f69-54vqt\" (UID: \"8e32da86-0b51-4502-b85d-73d1671c92f0\") " pod="kuadrant-system/authorino-557f876f69-54vqt"
Sep 10 23:07:46.337941 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.337909    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/8e32da86-0b51-4502-b85d-73d1671c92f0-kube-api-access-kfzc5\") pod \"authorino-557f876f69-54vqt\" (UID: \"8e32da86-0b51-4502-b85d-73d1671c92f0\") " pod="kuadrant-system/authorino-557f876f69-54vqt"
Sep 10 23:07:46.358252 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.358181    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-54vqt"]
Sep 10 23:07:46.358420 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.358408    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54vqt"
Sep 10 23:07:46.404214 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.404181    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-54ddf6b558-89khp"]
Sep 10 23:07:46.408244 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.408221    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54ddf6b558-89khp"
Sep 10 23:07:46.418702 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.418631    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-54ddf6b558-89khp"]
Sep 10 23:07:46.453967 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.453931    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:46.517287 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.517254    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54ddf6b558-89khp"]
Sep 10 23:07:46.517663 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:07:46.517481    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-v4768], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-54ddf6b558-89khp" podUID="6e02ebe3-7f72-414e-bb6d-c22fd2edbc18"
Sep 10 23:07:46.528859 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.528830    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4768\" (UniqueName: \"kubernetes.io/projected/6e02ebe3-7f72-414e-bb6d-c22fd2edbc18-kube-api-access-v4768\") pod \"authorino-54ddf6b558-89khp\" (UID: \"6e02ebe3-7f72-414e-bb6d-c22fd2edbc18\") " pod="kuadrant-system/authorino-54ddf6b558-89khp"
Sep 10 23:07:46.559062 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.559029    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-55767b5696-84t58"]
Sep 10 23:07:46.563223 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.563202    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:46.566132 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.566110    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 23:07:46.574086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.574062    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55767b5696-84t58"]
Sep 10 23:07:46.629752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.629684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/385cf433-09d6-42e8-a88a-94b9c55493db-tls-cert\") pod \"authorino-55767b5696-84t58\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") " pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:46.629752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.629727    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4768\" (UniqueName: \"kubernetes.io/projected/6e02ebe3-7f72-414e-bb6d-c22fd2edbc18-kube-api-access-v4768\") pod \"authorino-54ddf6b558-89khp\" (UID: \"6e02ebe3-7f72-414e-bb6d-c22fd2edbc18\") " pod="kuadrant-system/authorino-54ddf6b558-89khp"
Sep 10 23:07:46.629752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.629748    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfhkx\" (UniqueName: \"kubernetes.io/projected/385cf433-09d6-42e8-a88a-94b9c55493db-kube-api-access-hfhkx\") pod \"authorino-55767b5696-84t58\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") " pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:46.650900 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.650872    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-54vqt"]
Sep 10 23:07:46.654173 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:07:46.654145    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8e32da86_0b51_4502_b85d_73d1671c92f0.slice/crio-1c2f016d107478d926cbe0921e91c2b19db0bab37fe3ba8483023541595154c6 WatchSource:0}: Error finding container 1c2f016d107478d926cbe0921e91c2b19db0bab37fe3ba8483023541595154c6: Status 404 returned error can't find the container with id 1c2f016d107478d926cbe0921e91c2b19db0bab37fe3ba8483023541595154c6
Sep 10 23:07:46.654912 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.654888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4768\" (UniqueName: \"kubernetes.io/projected/6e02ebe3-7f72-414e-bb6d-c22fd2edbc18-kube-api-access-v4768\") pod \"authorino-54ddf6b558-89khp\" (UID: \"6e02ebe3-7f72-414e-bb6d-c22fd2edbc18\") " pod="kuadrant-system/authorino-54ddf6b558-89khp"
Sep 10 23:07:46.730659 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.730624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/385cf433-09d6-42e8-a88a-94b9c55493db-tls-cert\") pod \"authorino-55767b5696-84t58\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") " pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:46.730845 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.730667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfhkx\" (UniqueName: \"kubernetes.io/projected/385cf433-09d6-42e8-a88a-94b9c55493db-kube-api-access-hfhkx\") pod \"authorino-55767b5696-84t58\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") " pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:46.733099 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.733072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/385cf433-09d6-42e8-a88a-94b9c55493db-tls-cert\") pod \"authorino-55767b5696-84t58\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") " pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:46.740001 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.739982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfhkx\" (UniqueName: \"kubernetes.io/projected/385cf433-09d6-42e8-a88a-94b9c55493db-kube-api-access-hfhkx\") pod \"authorino-55767b5696-84t58\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") " pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:46.872605 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:46.872574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:07:47.158985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.158952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-55767b5696-84t58"]
Sep 10 23:07:47.162172 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:07:47.162144    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod385cf433_09d6_42e8_a88a_94b9c55493db.slice/crio-a8438e2e27ba20ccbea8fed201027ab6304f24ab2e1ee4f479067efb5eacb6da WatchSource:0}: Error finding container a8438e2e27ba20ccbea8fed201027ab6304f24ab2e1ee4f479067efb5eacb6da: Status 404 returned error can't find the container with id a8438e2e27ba20ccbea8fed201027ab6304f24ab2e1ee4f479067efb5eacb6da
Sep 10 23:07:47.512749 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.512667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55767b5696-84t58" event={"ID":"385cf433-09d6-42e8-a88a-94b9c55493db","Type":"ContainerStarted","Data":"a8438e2e27ba20ccbea8fed201027ab6304f24ab2e1ee4f479067efb5eacb6da"}
Sep 10 23:07:47.514131 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.514104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-54vqt" event={"ID":"8e32da86-0b51-4502-b85d-73d1671c92f0","Type":"ContainerStarted","Data":"2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180"}
Sep 10 23:07:47.514266 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.514148    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-54vqt" event={"ID":"8e32da86-0b51-4502-b85d-73d1671c92f0","Type":"ContainerStarted","Data":"1c2f016d107478d926cbe0921e91c2b19db0bab37fe3ba8483023541595154c6"}
Sep 10 23:07:47.514266 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.514112    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54ddf6b558-89khp"
Sep 10 23:07:47.514266 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.514233    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-557f876f69-54vqt" podUID="8e32da86-0b51-4502-b85d-73d1671c92f0" containerName="authorino" containerID="cri-o://2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180" gracePeriod=30
Sep 10 23:07:47.535729 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.535682    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-557f876f69-54vqt" podStartSLOduration=1.140765322 podStartE2EDuration="1.535665526s" totalImagesPullingTime="394.900204ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:07:47.533674242 +0000 UTC m=+583.969729015" watchObservedRunningTime="2026-09-10 23:07:47.535665526 +0000 UTC m=+583.971720299"
Sep 10 23:07:47.579818 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.579773    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54ddf6b558-89khp"
Sep 10 23:07:47.638830 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.638767    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v4768\" (UniqueName: \"kubernetes.io/projected/6e02ebe3-7f72-414e-bb6d-c22fd2edbc18-kube-api-access-v4768\") pod \"6e02ebe3-7f72-414e-bb6d-c22fd2edbc18\" (UID: \"6e02ebe3-7f72-414e-bb6d-c22fd2edbc18\") "
Sep 10 23:07:47.640922 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.640893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e02ebe3-7f72-414e-bb6d-c22fd2edbc18-kube-api-access-v4768" (OuterVolumeSpecName: "kube-api-access-v4768") pod "6e02ebe3-7f72-414e-bb6d-c22fd2edbc18" (UID: "6e02ebe3-7f72-414e-bb6d-c22fd2edbc18"). InnerVolumeSpecName "kube-api-access-v4768". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:07:47.740345 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.740311    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4768\" (UniqueName: \"kubernetes.io/projected/6e02ebe3-7f72-414e-bb6d-c22fd2edbc18-kube-api-access-v4768\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:07:47.784486 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.784461    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54vqt"
Sep 10 23:07:47.840978 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.840951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/8e32da86-0b51-4502-b85d-73d1671c92f0-kube-api-access-kfzc5\") pod \"8e32da86-0b51-4502-b85d-73d1671c92f0\" (UID: \"8e32da86-0b51-4502-b85d-73d1671c92f0\") "
Sep 10 23:07:47.842951 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.842926    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e32da86-0b51-4502-b85d-73d1671c92f0-kube-api-access-kfzc5" (OuterVolumeSpecName: "kube-api-access-kfzc5") pod "8e32da86-0b51-4502-b85d-73d1671c92f0" (UID: "8e32da86-0b51-4502-b85d-73d1671c92f0"). InnerVolumeSpecName "kube-api-access-kfzc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:07:47.942353 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:47.942321    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfzc5\" (UniqueName: \"kubernetes.io/projected/8e32da86-0b51-4502-b85d-73d1671c92f0-kube-api-access-kfzc5\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:07:48.520237 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.520193    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55767b5696-84t58" event={"ID":"385cf433-09d6-42e8-a88a-94b9c55493db","Type":"ContainerStarted","Data":"918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5"}
Sep 10 23:07:48.521548 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.521521    2577 generic.go:356] "Generic (PLEG): container finished" podID="8e32da86-0b51-4502-b85d-73d1671c92f0" containerID="2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180" exitCode=0
Sep 10 23:07:48.521663 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.521562    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-54vqt"
Sep 10 23:07:48.521663 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.521599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-54vqt" event={"ID":"8e32da86-0b51-4502-b85d-73d1671c92f0","Type":"ContainerDied","Data":"2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180"}
Sep 10 23:07:48.521663 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.521634    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-557f876f69-54vqt" event={"ID":"8e32da86-0b51-4502-b85d-73d1671c92f0","Type":"ContainerDied","Data":"1c2f016d107478d926cbe0921e91c2b19db0bab37fe3ba8483023541595154c6"}
Sep 10 23:07:48.521663 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.521656    2577 scope.go:117] "RemoveContainer" containerID="2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180"
Sep 10 23:07:48.521816 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.521747    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-54ddf6b558-89khp"
Sep 10 23:07:48.530136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.530116    2577 scope.go:117] "RemoveContainer" containerID="2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180"
Sep 10 23:07:48.530530 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:07:48.530507    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180\": container with ID starting with 2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180 not found: ID does not exist" containerID="2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180"
Sep 10 23:07:48.530628 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.530534    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180"} err="failed to get container status \"2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180\": rpc error: code = NotFound desc = could not find container \"2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180\": container with ID starting with 2fb074975ef515d301e97b9cce706316e714d1eb28ea75f376efa717bac92180 not found: ID does not exist"
Sep 10 23:07:48.540154 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.540114    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-55767b5696-84t58" podStartSLOduration=1.690499492 podStartE2EDuration="2.54010033s" totalImagesPullingTime="849.600838ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:07:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:07:48.538329706 +0000 UTC m=+584.974384480" watchObservedRunningTime="2026-09-10 23:07:48.54010033 +0000 UTC m=+584.976155103"
Sep 10 23:07:48.556875 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.556847    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-54vqt"]
Sep 10 23:07:48.565720 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.565693    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-54vqt"]
Sep 10 23:07:48.603587 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.603506    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-54ddf6b558-89khp"]
Sep 10 23:07:48.609136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:48.609107    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-54ddf6b558-89khp"]
Sep 10 23:07:50.214083 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:50.214052    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e02ebe3-7f72-414e-bb6d-c22fd2edbc18" path="/var/lib/kubelet/pods/6e02ebe3-7f72-414e-bb6d-c22fd2edbc18/volumes"
Sep 10 23:07:50.214544 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:50.214265    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e32da86-0b51-4502-b85d-73d1671c92f0" path="/var/lib/kubelet/pods/8e32da86-0b51-4502-b85d-73d1671c92f0/volumes"
Sep 10 23:07:53.698170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.698133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:53.738142 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.738107    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7769777fc8-6m62q"]
Sep 10 23:07:53.738471 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.738458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e32da86-0b51-4502-b85d-73d1671c92f0" containerName="authorino"
Sep 10 23:07:53.738519 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.738473    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e32da86-0b51-4502-b85d-73d1671c92f0" containerName="authorino"
Sep 10 23:07:53.738553 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.738536    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e32da86-0b51-4502-b85d-73d1671c92f0" containerName="authorino"
Sep 10 23:07:53.743652 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.743631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:53.746158 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.746130    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 23:07:53.746376 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.746361    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 23:07:53.746981 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.746957    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 23:07:53.747100 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.746957    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2npq7\""
Sep 10 23:07:53.755542 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.755514    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7769777fc8-6m62q"]
Sep 10 23:07:53.779457 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.779418    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:53.845565 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.845534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:53.897848 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.897806    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bmrc4\" (UniqueName: \"kubernetes.io/projected/162405ac-6419-4448-ade6-8fa78fae98c0-kube-api-access-bmrc4\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:53.898042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.897870    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-webhook-cert\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:53.898042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.897976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:53.933012 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.932972    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cf685499f-bjvqh"]
Sep 10 23:07:53.935742 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.935722    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:53.955680 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.955614    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cf685499f-bjvqh"]
Sep 10 23:07:53.999166 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.999126    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:53.999314 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.999195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bmrc4\" (UniqueName: \"kubernetes.io/projected/162405ac-6419-4448-ade6-8fa78fae98c0-kube-api-access-bmrc4\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:53.999314 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:53.999239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-webhook-cert\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:53.999314 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:07:53.999269    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-metrics-tls: secret "maas-controller-metrics-tls" not found
Sep 10 23:07:53.999465 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:07:53.999331    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs podName:162405ac-6419-4448-ade6-8fa78fae98c0 nodeName:}" failed. No retries permitted until 2026-09-10 23:07:54.49931365 +0000 UTC m=+590.935368406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs") pod "maas-controller-7769777fc8-6m62q" (UID: "162405ac-6419-4448-ade6-8fa78fae98c0") : secret "maas-controller-metrics-tls" not found
Sep 10 23:07:54.001754 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.001725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-webhook-cert\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:54.007587 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.007559    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:54.011434 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.011412    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bmrc4\" (UniqueName: \"kubernetes.io/projected/162405ac-6419-4448-ade6-8fa78fae98c0-kube-api-access-bmrc4\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:54.084319 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.084288    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:54.100645 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.100615    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-metrics-certs\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.100833 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.100707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4frj9\" (UniqueName: \"kubernetes.io/projected/7af435e9-16a6-4feb-8649-3ddf1af20c8d-kube-api-access-4frj9\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.100833 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.100735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-webhook-cert\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.201531 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.201496    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4frj9\" (UniqueName: \"kubernetes.io/projected/7af435e9-16a6-4feb-8649-3ddf1af20c8d-kube-api-access-4frj9\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.201739 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.201550    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-webhook-cert\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.201739 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.201596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-metrics-certs\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.204060 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.204034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-webhook-cert\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.204226 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.204203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-metrics-certs\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.215631 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.215569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4frj9\" (UniqueName: \"kubernetes.io/projected/7af435e9-16a6-4feb-8649-3ddf1af20c8d-kube-api-access-4frj9\") pod \"maas-controller-cf685499f-bjvqh\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") " pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.251591 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.251543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:54.504203 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.504118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:54.506696 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.506672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs\") pod \"maas-controller-7769777fc8-6m62q\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") " pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:54.527282 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.527252    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cf685499f-bjvqh"]
Sep 10 23:07:54.530007 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:07:54.529985    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7af435e9_16a6_4feb_8649_3ddf1af20c8d.slice/crio-3ee0c12de83855765919c8c2d9dcfabcb1956e3a33c8d6e18135b09ecb585213 WatchSource:0}: Error finding container 3ee0c12de83855765919c8c2d9dcfabcb1956e3a33c8d6e18135b09ecb585213: Status 404 returned error can't find the container with id 3ee0c12de83855765919c8c2d9dcfabcb1956e3a33c8d6e18135b09ecb585213
Sep 10 23:07:54.549527 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.549497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-bjvqh" event={"ID":"7af435e9-16a6-4feb-8649-3ddf1af20c8d","Type":"ContainerStarted","Data":"3ee0c12de83855765919c8c2d9dcfabcb1956e3a33c8d6e18135b09ecb585213"}
Sep 10 23:07:54.655831 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.655769    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:54.932721 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:54.932629    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7769777fc8-6m62q"]
Sep 10 23:07:54.936400 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:07:54.936375    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod162405ac_6419_4448_ade6_8fa78fae98c0.slice/crio-48b663784bb1105146e474736cc442423a8e7a0dce9ad03cf04d3f6e13329371 WatchSource:0}: Error finding container 48b663784bb1105146e474736cc442423a8e7a0dce9ad03cf04d3f6e13329371: Status 404 returned error can't find the container with id 48b663784bb1105146e474736cc442423a8e7a0dce9ad03cf04d3f6e13329371
Sep 10 23:07:55.558357 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:55.558309    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7769777fc8-6m62q" event={"ID":"162405ac-6419-4448-ade6-8fa78fae98c0","Type":"ContainerStarted","Data":"48b663784bb1105146e474736cc442423a8e7a0dce9ad03cf04d3f6e13329371"}
Sep 10 23:07:57.262927 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.262880    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:57.368005 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.367928    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:07:57.569272 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.569239    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-bjvqh" event={"ID":"7af435e9-16a6-4feb-8649-3ddf1af20c8d","Type":"ContainerStarted","Data":"68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561"}
Sep 10 23:07:57.569448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.569304    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:07:57.570591 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.570564    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7769777fc8-6m62q" event={"ID":"162405ac-6419-4448-ade6-8fa78fae98c0","Type":"ContainerStarted","Data":"e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc"}
Sep 10 23:07:57.570697 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.570620    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:07:57.603098 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.603037    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cf685499f-bjvqh" podStartSLOduration=2.06863455 podStartE2EDuration="4.603016515s" totalImagesPullingTime="2.534381965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:07:57.59958203 +0000 UTC m=+594.035636803" watchObservedRunningTime="2026-09-10 23:07:57.603016515 +0000 UTC m=+594.039071287"
Sep 10 23:07:57.633854 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:07:57.633711    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-7769777fc8-6m62q" podStartSLOduration=2.49763313 podStartE2EDuration="4.633691984s" totalImagesPullingTime="2.136058854s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:07:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:07:57.632412503 +0000 UTC m=+594.068467275" watchObservedRunningTime="2026-09-10 23:07:57.633691984 +0000 UTC m=+594.069746755"
Sep 10 23:08:08.577901 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.577867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:08:08.578373 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.577939    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:08:08.664336 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.664300    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7769777fc8-6m62q"]
Sep 10 23:08:08.664604 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.664547    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-7769777fc8-6m62q" podUID="162405ac-6419-4448-ade6-8fa78fae98c0" containerName="manager" containerID="cri-o://e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc" gracePeriod=30
Sep 10 23:08:08.906057 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.906032    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:08:08.931402 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.931368    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-webhook-cert\") pod \"162405ac-6419-4448-ade6-8fa78fae98c0\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") "
Sep 10 23:08:08.931590 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.931451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bmrc4\" (UniqueName: \"kubernetes.io/projected/162405ac-6419-4448-ade6-8fa78fae98c0-kube-api-access-bmrc4\") pod \"162405ac-6419-4448-ade6-8fa78fae98c0\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") "
Sep 10 23:08:08.931929 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.931903    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs\") pod \"162405ac-6419-4448-ade6-8fa78fae98c0\" (UID: \"162405ac-6419-4448-ade6-8fa78fae98c0\") "
Sep 10 23:08:08.934077 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.934046    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "162405ac-6419-4448-ade6-8fa78fae98c0" (UID: "162405ac-6419-4448-ade6-8fa78fae98c0"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:08:08.934179 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.934148    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "162405ac-6419-4448-ade6-8fa78fae98c0" (UID: "162405ac-6419-4448-ade6-8fa78fae98c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:08:08.934604 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:08.934573    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/162405ac-6419-4448-ade6-8fa78fae98c0-kube-api-access-bmrc4" (OuterVolumeSpecName: "kube-api-access-bmrc4") pod "162405ac-6419-4448-ade6-8fa78fae98c0" (UID: "162405ac-6419-4448-ade6-8fa78fae98c0"). InnerVolumeSpecName "kube-api-access-bmrc4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:08:09.032897 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.032856    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-webhook-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:08:09.032897 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.032889    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bmrc4\" (UniqueName: \"kubernetes.io/projected/162405ac-6419-4448-ade6-8fa78fae98c0-kube-api-access-bmrc4\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:08:09.032897 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.032899    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/162405ac-6419-4448-ade6-8fa78fae98c0-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:08:09.633100 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.633066    2577 generic.go:356] "Generic (PLEG): container finished" podID="162405ac-6419-4448-ade6-8fa78fae98c0" containerID="e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc" exitCode=0
Sep 10 23:08:09.633499 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.633110    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7769777fc8-6m62q"
Sep 10 23:08:09.633499 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.633114    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7769777fc8-6m62q" event={"ID":"162405ac-6419-4448-ade6-8fa78fae98c0","Type":"ContainerDied","Data":"e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc"}
Sep 10 23:08:09.633499 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.633220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7769777fc8-6m62q" event={"ID":"162405ac-6419-4448-ade6-8fa78fae98c0","Type":"ContainerDied","Data":"48b663784bb1105146e474736cc442423a8e7a0dce9ad03cf04d3f6e13329371"}
Sep 10 23:08:09.633499 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.633241    2577 scope.go:117] "RemoveContainer" containerID="e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc"
Sep 10 23:08:09.642957 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.642937    2577 scope.go:117] "RemoveContainer" containerID="e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc"
Sep 10 23:08:09.643244 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:08:09.643224    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc\": container with ID starting with e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc not found: ID does not exist" containerID="e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc"
Sep 10 23:08:09.643314 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.643251    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc"} err="failed to get container status \"e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc\": rpc error: code = NotFound desc = could not find container \"e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc\": container with ID starting with e0c60319d6fa55c7f73db40f0e8adadb9a0aedc8e636a43008b50d1d6e8a13fc not found: ID does not exist"
Sep 10 23:08:09.657083 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.657055    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7769777fc8-6m62q"]
Sep 10 23:08:09.661258 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:09.661235    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7769777fc8-6m62q"]
Sep 10 23:08:10.214298 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:10.214265    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="162405ac-6419-4448-ade6-8fa78fae98c0" path="/var/lib/kubelet/pods/162405ac-6419-4448-ade6-8fa78fae98c0/volumes"
Sep 10 23:08:11.378017 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:11.377982    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:27.947047 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:27.946968    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:28.057291 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:28.057257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:29.408726 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.408684    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:29.493493 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.493463    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:29.592077 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.592036    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:29.667958 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.667881    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"]
Sep 10 23:08:29.668213 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.668201    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="162405ac-6419-4448-ade6-8fa78fae98c0" containerName="manager"
Sep 10 23:08:29.668213 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.668213    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="162405ac-6419-4448-ade6-8fa78fae98c0" containerName="manager"
Sep 10 23:08:29.668288 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.668272    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="162405ac-6419-4448-ade6-8fa78fae98c0" containerName="manager"
Sep 10 23:08:29.671107 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.671088    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.673421 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.673393    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 23:08:29.674173 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.674154    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-9dl9w\""
Sep 10 23:08:29.674256 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.674189    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 23:08:29.684216 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.684193    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"]
Sep 10 23:08:29.747253 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.747218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:29.808209 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.808178    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-maas-api-tls\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.808375 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.808233    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-metrics-certs\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.808375 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.808296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110f6dd0-4140-4c6c-9653-9b5d5415bb16-openshift-service-ca\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.808375 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.808332    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxrzt\" (UniqueName: \"kubernetes.io/projected/110f6dd0-4140-4c6c-9653-9b5d5415bb16-kube-api-access-vxrzt\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.909432 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.909390    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-maas-api-tls\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.909620 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.909523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-metrics-certs\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.909620 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.909556    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110f6dd0-4140-4c6c-9653-9b5d5415bb16-openshift-service-ca\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.909620 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.909593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxrzt\" (UniqueName: \"kubernetes.io/projected/110f6dd0-4140-4c6c-9653-9b5d5415bb16-kube-api-access-vxrzt\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.910322 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.910295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110f6dd0-4140-4c6c-9653-9b5d5415bb16-openshift-service-ca\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.912091 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.912057    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-metrics-certs\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.912198 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.912086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-maas-api-tls\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.921714 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.921626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxrzt\" (UniqueName: \"kubernetes.io/projected/110f6dd0-4140-4c6c-9653-9b5d5415bb16-kube-api-access-vxrzt\") pod \"maas-api-7f7f8945f9-45hfg\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") " pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:29.981110 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:29.981071    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:30.143205 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.143175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:30.247485 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.247399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:30.255963 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.255938    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"]
Sep 10 23:08:30.258136 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:08:30.258108    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod110f6dd0_4140_4c6c_9653_9b5d5415bb16.slice/crio-2d5fcbc9e45c78446fec1234c7368167c629ecf96e28d398f14741a6c5605375 WatchSource:0}: Error finding container 2d5fcbc9e45c78446fec1234c7368167c629ecf96e28d398f14741a6c5605375: Status 404 returned error can't find the container with id 2d5fcbc9e45c78446fec1234c7368167c629ecf96e28d398f14741a6c5605375
Sep 10 23:08:30.337981 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.337946    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"]
Sep 10 23:08:30.342960 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.342929    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.345911 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.345885    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-prl7r\""
Sep 10 23:08:30.346226 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.346207    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 23:08:30.356662 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.356609    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"]
Sep 10 23:08:30.412686 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.412645    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4h7\" (UniqueName: \"kubernetes.io/projected/ca8081b0-eff4-4497-8af2-eaaf215d27a1-kube-api-access-lj4h7\") pod \"payload-pre-processing-59fbc47bf8-qxh2c\" (UID: \"ca8081b0-eff4-4497-8af2-eaaf215d27a1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.413094 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.412722    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8081b0-eff4-4497-8af2-eaaf215d27a1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qxh2c\" (UID: \"ca8081b0-eff4-4497-8af2-eaaf215d27a1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.514157 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.513561    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8081b0-eff4-4497-8af2-eaaf215d27a1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qxh2c\" (UID: \"ca8081b0-eff4-4497-8af2-eaaf215d27a1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.514157 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.513673    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4h7\" (UniqueName: \"kubernetes.io/projected/ca8081b0-eff4-4497-8af2-eaaf215d27a1-kube-api-access-lj4h7\") pod \"payload-pre-processing-59fbc47bf8-qxh2c\" (UID: \"ca8081b0-eff4-4497-8af2-eaaf215d27a1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.514534 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.514510    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ca8081b0-eff4-4497-8af2-eaaf215d27a1-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-qxh2c\" (UID: \"ca8081b0-eff4-4497-8af2-eaaf215d27a1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.530028 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.529998    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4h7\" (UniqueName: \"kubernetes.io/projected/ca8081b0-eff4-4497-8af2-eaaf215d27a1-kube-api-access-lj4h7\") pod \"payload-pre-processing-59fbc47bf8-qxh2c\" (UID: \"ca8081b0-eff4-4497-8af2-eaaf215d27a1\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.654093 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.654052    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:30.732921 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.732843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg" event={"ID":"110f6dd0-4140-4c6c-9653-9b5d5415bb16","Type":"ContainerStarted","Data":"2d5fcbc9e45c78446fec1234c7368167c629ecf96e28d398f14741a6c5605375"}
Sep 10 23:08:30.955430 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:30.955399    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"]
Sep 10 23:08:30.958349 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:08:30.958317    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca8081b0_eff4_4497_8af2_eaaf215d27a1.slice/crio-3db6e3d220096ea0bd58848c877aedab04b22e9805ce7c846a35dd4eb0acf73a WatchSource:0}: Error finding container 3db6e3d220096ea0bd58848c877aedab04b22e9805ce7c846a35dd4eb0acf73a: Status 404 returned error can't find the container with id 3db6e3d220096ea0bd58848c877aedab04b22e9805ce7c846a35dd4eb0acf73a
Sep 10 23:08:31.747217 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:31.747175    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c" event={"ID":"ca8081b0-eff4-4497-8af2-eaaf215d27a1","Type":"ContainerStarted","Data":"3db6e3d220096ea0bd58848c877aedab04b22e9805ce7c846a35dd4eb0acf73a"}
Sep 10 23:08:32.754180 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:32.754138    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c" event={"ID":"ca8081b0-eff4-4497-8af2-eaaf215d27a1","Type":"ContainerStarted","Data":"7a48a9b7a852644266dd2c71c174bc5856a4fc89dc09780567e006fb28a96b65"}
Sep 10 23:08:32.754646 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:32.754254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:32.755812 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:32.755772    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg" event={"ID":"110f6dd0-4140-4c6c-9653-9b5d5415bb16","Type":"ContainerStarted","Data":"6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64"}
Sep 10 23:08:32.755937 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:32.755900    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:32.775985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:32.775929    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c" podStartSLOduration=1.153465276 podStartE2EDuration="2.775913154s" totalImagesPullingTime="1.622447878s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:32.774513833 +0000 UTC m=+629.210568605" watchObservedRunningTime="2026-09-10 23:08:32.775913154 +0000 UTC m=+629.211967928"
Sep 10 23:08:32.798304 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:32.798247    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg" podStartSLOduration=2.177410223 podStartE2EDuration="3.798227134s" totalImagesPullingTime="1.620816911s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:32.797874322 +0000 UTC m=+629.233929107" watchObservedRunningTime="2026-09-10 23:08:32.798227134 +0000 UTC m=+629.234281907"
Sep 10 23:08:38.765547 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:38.765513    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:08:40.499198 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:40.499154    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:40.590231 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:40.590196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:43.763183 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:43.763155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-qxh2c"
Sep 10 23:08:51.690429 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:51.690384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:58.047214 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.047180    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"]
Sep 10 23:08:58.050636 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.050609    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.052264 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.052231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs7vj\" (UniqueName: \"kubernetes.io/projected/6ba46953-bba7-4684-885e-d1f21261e9bb-kube-api-access-gs7vj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.052395 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.052299    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.052395 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.052333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba46953-bba7-4684-885e-d1f21261e9bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.052522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.052400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.052522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.052463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.052620 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.052559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.054452 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.054433    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 23:08:58.054564 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.054435    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-85z7r\""
Sep 10 23:08:58.054930 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.054910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 23:08:58.055152 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.054959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 23:08:58.071310 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.071284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"]
Sep 10 23:08:58.084427 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.084403    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:08:58.153450 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.153417    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.153642 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.153456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba46953-bba7-4684-885e-d1f21261e9bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.153642 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.153479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.153642 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.153502    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.153642 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.153547    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.153642 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.153571    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs7vj\" (UniqueName: \"kubernetes.io/projected/6ba46953-bba7-4684-885e-d1f21261e9bb-kube-api-access-gs7vj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.154035 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.154010    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.154078 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.154035    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.154164 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.154141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.155732 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.155714    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6ba46953-bba7-4684-885e-d1f21261e9bb-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.156028 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.156012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6ba46953-bba7-4684-885e-d1f21261e9bb-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.163420 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.163398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs7vj\" (UniqueName: \"kubernetes.io/projected/6ba46953-bba7-4684-885e-d1f21261e9bb-kube-api-access-gs7vj\") pod \"e2e-distinct-simulated-kserve-59975d69f9-nzffn\" (UID: \"6ba46953-bba7-4684-885e-d1f21261e9bb\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.362454 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.362353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:08:58.639525 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.639500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"]
Sep 10 23:08:58.641637 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:08:58.641606    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ba46953_bba7_4684_885e_d1f21261e9bb.slice/crio-630408145efd6b78d2f8461357fa685a9bb772dd240d49c150fd737530325285 WatchSource:0}: Error finding container 630408145efd6b78d2f8461357fa685a9bb772dd240d49c150fd737530325285: Status 404 returned error can't find the container with id 630408145efd6b78d2f8461357fa685a9bb772dd240d49c150fd737530325285
Sep 10 23:08:58.873065 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:58.873019    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn" event={"ID":"6ba46953-bba7-4684-885e-d1f21261e9bb","Type":"ContainerStarted","Data":"630408145efd6b78d2f8461357fa685a9bb772dd240d49c150fd737530325285"}
Sep 10 23:08:59.878971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:59.878938    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn" event={"ID":"6ba46953-bba7-4684-885e-d1f21261e9bb","Type":"ContainerStarted","Data":"aafc2fe3c42158eea7ce145969b8b89c040b7f160d930f522731cdfa71906575"}
Sep 10 23:08:59.900444 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:08:59.900390    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn" podStartSLOduration=0.884716909 podStartE2EDuration="1.900373351s" totalImagesPullingTime="1.015656442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:08:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:08:59.899536549 +0000 UTC m=+656.335591322" watchObservedRunningTime="2026-09-10 23:08:59.900373351 +0000 UTC m=+656.336428123"
Sep 10 23:09:02.462404 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:02.462365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:08.362832 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:08.362770    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:09:08.363223 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:08.362867    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:09:08.375182 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:08.375145    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:09:08.928420 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:08.928388    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-nzffn"
Sep 10 23:09:12.185006 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.184970    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"]
Sep 10 23:09:12.189488 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.189463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.191937 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.191911    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 23:09:12.202035 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.202006    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"]
Sep 10 23:09:12.203586 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.203554    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:12.274563 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.274526    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-kube-api-access-8m57x\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.274563 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.274583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.274836 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.274649    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.274836 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.274690    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.274836 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.274763    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.274967 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.274943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376005 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.375968    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376018    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-kube-api-access-8m57x\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376144    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376554 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376521    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376677 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.376677 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.376588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.378448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.378421    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.378655 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.378636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.387759 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.387730    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m57x\" (UniqueName: \"kubernetes.io/projected/982bf7c2-483b-4c59-9f2a-9c4a0ca49773-kube-api-access-8m57x\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm\" (UID: \"982bf7c2-483b-4c59-9f2a-9c4a0ca49773\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.499220 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.499121    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:12.780348 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.780269    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"]
Sep 10 23:09:12.783396 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:09:12.783371    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod982bf7c2_483b_4c59_9f2a_9c4a0ca49773.slice/crio-6ed4375d5c9fbdb82f9b1890f16eb856025805d27f4881454ad6b021e03f0063 WatchSource:0}: Error finding container 6ed4375d5c9fbdb82f9b1890f16eb856025805d27f4881454ad6b021e03f0063: Status 404 returned error can't find the container with id 6ed4375d5c9fbdb82f9b1890f16eb856025805d27f4881454ad6b021e03f0063
Sep 10 23:09:12.784672 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.784655    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:09:12.936043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:12.936003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm" event={"ID":"982bf7c2-483b-4c59-9f2a-9c4a0ca49773","Type":"ContainerStarted","Data":"6ed4375d5c9fbdb82f9b1890f16eb856025805d27f4881454ad6b021e03f0063"}
Sep 10 23:09:13.943434 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:13.943395    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm" event={"ID":"982bf7c2-483b-4c59-9f2a-9c4a0ca49773","Type":"ContainerStarted","Data":"e45d33704e886c912ad97808f0a644f717bbad798c4c0e925e2df7cf237990c1"}
Sep 10 23:09:13.976570 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:13.976506    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm" podStartSLOduration=1.799796889 podStartE2EDuration="1.976489487s" totalImagesPullingTime="176.692598ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:13.975301413 +0000 UTC m=+670.411356188" watchObservedRunningTime="2026-09-10 23:09:13.976489487 +0000 UTC m=+670.412544260"
Sep 10 23:09:16.418150 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:16.418114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:22.499560 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:22.499514    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:22.500046 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:22.499591    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:22.512249 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:22.512218    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:22.995307 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:22.995279    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm"
Sep 10 23:09:24.775160 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.775128    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"]
Sep 10 23:09:24.778836 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.778812    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.781442 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.781416    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 23:09:24.791523 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.791500    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"]
Sep 10 23:09:24.815746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.815712    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:24.893461 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.893420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.893637 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.893489    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.893637 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.893515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.893637 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.893537    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwmdv\" (UniqueName: \"kubernetes.io/projected/2bbbadb5-1335-448c-ae13-7798c53081e0-kube-api-access-lwmdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.893760 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.893632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbbadb5-1335-448c-ae13-7798c53081e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.893760 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.893660    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995051 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995068    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995221    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwmdv\" (UniqueName: \"kubernetes.io/projected/2bbbadb5-1335-448c-ae13-7798c53081e0-kube-api-access-lwmdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995337 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbbadb5-1335-448c-ae13-7798c53081e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995337 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995412 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995391    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995542 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.995701 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.995613    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.997342 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.997317    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2bbbadb5-1335-448c-ae13-7798c53081e0-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:24.997802 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:24.997758    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2bbbadb5-1335-448c-ae13-7798c53081e0-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:25.006472 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:25.006442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwmdv\" (UniqueName: \"kubernetes.io/projected/2bbbadb5-1335-448c-ae13-7798c53081e0-kube-api-access-lwmdv\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8\" (UID: \"2bbbadb5-1335-448c-ae13-7798c53081e0\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:25.090698 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:25.090602    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:25.372299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:25.372237    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"]
Sep 10 23:09:25.374535 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:09:25.374508    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bbbadb5_1335_448c_ae13_7798c53081e0.slice/crio-abd4f9188e7cc768880a3c0d4079dc33cca4384797f875bcab3134ad5f6ed339 WatchSource:0}: Error finding container abd4f9188e7cc768880a3c0d4079dc33cca4384797f875bcab3134ad5f6ed339: Status 404 returned error can't find the container with id abd4f9188e7cc768880a3c0d4079dc33cca4384797f875bcab3134ad5f6ed339
Sep 10 23:09:26.005235 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:26.005201    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8" event={"ID":"2bbbadb5-1335-448c-ae13-7798c53081e0","Type":"ContainerStarted","Data":"dac4b040d46311b4e47f666061948a781c41491de44055d8689ae487adbaca9b"}
Sep 10 23:09:26.005605 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:26.005275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8" event={"ID":"2bbbadb5-1335-448c-ae13-7798c53081e0","Type":"ContainerStarted","Data":"abd4f9188e7cc768880a3c0d4079dc33cca4384797f875bcab3134ad5f6ed339"}
Sep 10 23:09:26.032545 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:26.032482    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8" podStartSLOduration=1.858950934 podStartE2EDuration="2.032466991s" totalImagesPullingTime="173.516057ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:09:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:09:26.030353214 +0000 UTC m=+682.466407999" watchObservedRunningTime="2026-09-10 23:09:26.032466991 +0000 UTC m=+682.468521763"
Sep 10 23:09:30.807670 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:30.807636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:09:35.091398 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:35.091355    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:35.091835 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:35.091422    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:35.104832 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:35.104802    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:36.065603 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:36.065567    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8"
Sep 10 23:09:39.869659 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:09:39.869618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:27.052746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:27.052647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:27.140594 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:27.140556    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:40.125165 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.125121    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55767b5696-84t58"]
Sep 10 23:11:40.125641 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.125361    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-55767b5696-84t58" podUID="385cf433-09d6-42e8-a88a-94b9c55493db" containerName="authorino" containerID="cri-o://918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5" gracePeriod=30
Sep 10 23:11:40.372901 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.372868    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:11:40.485743 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.485711    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/385cf433-09d6-42e8-a88a-94b9c55493db-tls-cert\") pod \"385cf433-09d6-42e8-a88a-94b9c55493db\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") "
Sep 10 23:11:40.485948 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.485855    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfhkx\" (UniqueName: \"kubernetes.io/projected/385cf433-09d6-42e8-a88a-94b9c55493db-kube-api-access-hfhkx\") pod \"385cf433-09d6-42e8-a88a-94b9c55493db\" (UID: \"385cf433-09d6-42e8-a88a-94b9c55493db\") "
Sep 10 23:11:40.487878 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.487842    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385cf433-09d6-42e8-a88a-94b9c55493db-kube-api-access-hfhkx" (OuterVolumeSpecName: "kube-api-access-hfhkx") pod "385cf433-09d6-42e8-a88a-94b9c55493db" (UID: "385cf433-09d6-42e8-a88a-94b9c55493db"). InnerVolumeSpecName "kube-api-access-hfhkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:11:40.495721 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.495694    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385cf433-09d6-42e8-a88a-94b9c55493db-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "385cf433-09d6-42e8-a88a-94b9c55493db" (UID: "385cf433-09d6-42e8-a88a-94b9c55493db"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:11:40.586741 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.586698    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfhkx\" (UniqueName: \"kubernetes.io/projected/385cf433-09d6-42e8-a88a-94b9c55493db-kube-api-access-hfhkx\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:11:40.586741 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.586735    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/385cf433-09d6-42e8-a88a-94b9c55493db-tls-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:11:40.619214 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.619179    2577 generic.go:356] "Generic (PLEG): container finished" podID="385cf433-09d6-42e8-a88a-94b9c55493db" containerID="918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5" exitCode=0
Sep 10 23:11:40.619389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.619245    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-55767b5696-84t58"
Sep 10 23:11:40.619389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.619252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55767b5696-84t58" event={"ID":"385cf433-09d6-42e8-a88a-94b9c55493db","Type":"ContainerDied","Data":"918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5"}
Sep 10 23:11:40.619389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.619346    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-55767b5696-84t58" event={"ID":"385cf433-09d6-42e8-a88a-94b9c55493db","Type":"ContainerDied","Data":"a8438e2e27ba20ccbea8fed201027ab6304f24ab2e1ee4f479067efb5eacb6da"}
Sep 10 23:11:40.619389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.619363    2577 scope.go:117] "RemoveContainer" containerID="918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5"
Sep 10 23:11:40.629696 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.629676    2577 scope.go:117] "RemoveContainer" containerID="918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5"
Sep 10 23:11:40.630051 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:11:40.630029    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5\": container with ID starting with 918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5 not found: ID does not exist" containerID="918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5"
Sep 10 23:11:40.630134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.630065    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5"} err="failed to get container status \"918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5\": rpc error: code = NotFound desc = could not find container \"918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5\": container with ID starting with 918e52559738d095631725f44b54cd5d749422a900281f61a64417ab6f6f3af5 not found: ID does not exist"
Sep 10 23:11:40.656754 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.656720    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-55767b5696-84t58"]
Sep 10 23:11:40.680319 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.680289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:40.680486 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.680365    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-55767b5696-84t58"]
Sep 10 23:11:40.761230 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:40.761141    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:41.794378 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:41.794338    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:41.892635 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:41.892597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:42.214386 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:42.214350    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385cf433-09d6-42e8-a88a-94b9c55493db" path="/var/lib/kubelet/pods/385cf433-09d6-42e8-a88a-94b9c55493db/volumes"
Sep 10 23:11:48.376186 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:48.376152    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:57.660867 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:57.660820    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:57.756445 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:57.756411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:57.991351 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:57.991239    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:58.462011 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:58.461976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:59.109658 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:59.109629    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:59.210965 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:59.210934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:11:59.589806 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:11:59.589747    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:01.487312 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:01.487276    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:01.569973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:01.569938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:02.804749 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.804714    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"]
Sep 10 23:12:02.805308 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.805286    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385cf433-09d6-42e8-a88a-94b9c55493db" containerName="authorino"
Sep 10 23:12:02.805308 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.805308    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="385cf433-09d6-42e8-a88a-94b9c55493db" containerName="authorino"
Sep 10 23:12:02.805492 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.805430    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="385cf433-09d6-42e8-a88a-94b9c55493db" containerName="authorino"
Sep 10 23:12:02.807601 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.807583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.810200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.810172    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-28dbae-serving-cert\""
Sep 10 23:12:02.822364 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.822340    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"]
Sep 10 23:12:02.832240 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.832208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:02.882170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.882132    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-922dr\" (UniqueName: \"kubernetes.io/projected/4e9a3890-94f6-4b89-8b4b-805a9585cf12-kube-api-access-922dr\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.882377 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.882251    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9a3890-94f6-4b89-8b4b-805a9585cf12-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.882377 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.882336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.882377 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.882368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-metrics-certs\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.938908 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.938874    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:02.983082 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.983038    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9a3890-94f6-4b89-8b4b-805a9585cf12-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.983276 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.983104    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.983276 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.983152    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-metrics-certs\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.983276 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.983193    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-922dr\" (UniqueName: \"kubernetes.io/projected/4e9a3890-94f6-4b89-8b4b-805a9585cf12-kube-api-access-922dr\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.983276 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:02.983263    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-serving-cert: secret "maas-api-e2e-inf-a-28dbae-serving-cert" not found
Sep 10 23:12:02.983496 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:02.983353    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls podName:4e9a3890-94f6-4b89-8b4b-805a9585cf12 nodeName:}" failed. No retries permitted until 2026-09-10 23:12:03.483330798 +0000 UTC m=+839.919385563 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls") pod "maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw" (UID: "4e9a3890-94f6-4b89-8b4b-805a9585cf12") : secret "maas-api-e2e-inf-a-28dbae-serving-cert" not found
Sep 10 23:12:02.983847 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.983817    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9a3890-94f6-4b89-8b4b-805a9585cf12-openshift-service-ca\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.985710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.985682    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-metrics-certs\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:02.993847 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:02.993818    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-922dr\" (UniqueName: \"kubernetes.io/projected/4e9a3890-94f6-4b89-8b4b-805a9585cf12-kube-api-access-922dr\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:03.188916 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:03.188862    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:03.470835 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:03.470707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:03.488923 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:03.488889    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:03.491842 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:03.491813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls\") pod \"maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:03.590423 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:03.590384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:03.718456 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:03.718425    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:04.001504 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:04.001428    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"]
Sep 10 23:12:04.005331 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:04.005302    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e9a3890_94f6_4b89_8b4b_805a9585cf12.slice/crio-3cf5d299f00db3e1ae5814e57b4c54bc75935d1606e00be67de4ebb25cccdec0 WatchSource:0}: Error finding container 3cf5d299f00db3e1ae5814e57b4c54bc75935d1606e00be67de4ebb25cccdec0: Status 404 returned error can't find the container with id 3cf5d299f00db3e1ae5814e57b4c54bc75935d1606e00be67de4ebb25cccdec0
Sep 10 23:12:04.728295 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:04.728228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw" event={"ID":"4e9a3890-94f6-4b89-8b4b-805a9585cf12","Type":"ContainerStarted","Data":"3cf5d299f00db3e1ae5814e57b4c54bc75935d1606e00be67de4ebb25cccdec0"}
Sep 10 23:12:05.734084 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:05.734047    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw" event={"ID":"4e9a3890-94f6-4b89-8b4b-805a9585cf12","Type":"ContainerStarted","Data":"b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4"}
Sep 10 23:12:05.734472 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:05.734264    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:05.780801 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:05.780725    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw" podStartSLOduration=3.057082033 podStartE2EDuration="3.780706856s" totalImagesPullingTime="723.624823ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:05.77920929 +0000 UTC m=+842.215264065" watchObservedRunningTime="2026-09-10 23:12:05.780706856 +0000 UTC m=+842.216761622"
Sep 10 23:12:06.554159 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:06.554123    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:06.611945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:06.611907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:06.782356 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:06.782321    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:07.110674 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:07.110638    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:07.485702 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:07.485666    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:09.937625 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:09.937555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:10.029131 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:10.029090    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:11.102069 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.102035    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"]
Sep 10 23:12:11.105217 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.105192    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.108555 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.108380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-e7db0e72-serving-cert\""
Sep 10 23:12:11.119005 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.118976    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"]
Sep 10 23:12:11.140414 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.140385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:11.238187 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.238142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:11.259847 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.259812    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2a73ba-c225-41c9-add6-a6efc03748bf-openshift-service-ca\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.260043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.259890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.260043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.259916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-metrics-certs\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.260043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.259965    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sqsw\" (UniqueName: \"kubernetes.io/projected/9d2a73ba-c225-41c9-add6-a6efc03748bf-kube-api-access-9sqsw\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.361113 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.361009    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.361113 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.361063    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-metrics-certs\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.361355 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.361120    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sqsw\" (UniqueName: \"kubernetes.io/projected/9d2a73ba-c225-41c9-add6-a6efc03748bf-kube-api-access-9sqsw\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.361355 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.361176    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2a73ba-c225-41c9-add6-a6efc03748bf-openshift-service-ca\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.361355 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:11.361192    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-serving-cert: secret "maas-api-e2e-ait-e7db0e72-serving-cert" not found
Sep 10 23:12:11.361355 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:11.361268    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls podName:9d2a73ba-c225-41c9-add6-a6efc03748bf nodeName:}" failed. No retries permitted until 2026-09-10 23:12:11.861246882 +0000 UTC m=+848.297301636 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls") pod "maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws" (UID: "9d2a73ba-c225-41c9-add6-a6efc03748bf") : secret "maas-api-e2e-ait-e7db0e72-serving-cert" not found
Sep 10 23:12:11.361956 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.361933    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2a73ba-c225-41c9-add6-a6efc03748bf-openshift-service-ca\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.363643 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.363621    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-metrics-certs\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.381448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.381420    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sqsw\" (UniqueName: \"kubernetes.io/projected/9d2a73ba-c225-41c9-add6-a6efc03748bf-kube-api-access-9sqsw\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.625430 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.625335    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:11.744922 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.744889    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:12:11.772140 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.772106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:11.866988 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.866954    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.868531 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.868487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:11.870049 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.870027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls\") pod \"maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:11.973980 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:11.973938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:12.017793 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.017752    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:12.038273 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.035155    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"]
Sep 10 23:12:12.040035 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.040013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.043380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.043354    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-e7db0e72\""
Sep 10 23:12:12.047016 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.046992    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"]
Sep 10 23:12:12.047857 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.047839    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-e7db0e72-dockercfg-v8nmv\""
Sep 10 23:12:12.110010 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.109970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:12.171046 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.171002    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7dtt\" (UniqueName: \"kubernetes.io/projected/70e855fb-c3c6-48aa-aab1-6f2a180cb801-kube-api-access-d7dtt\") pod \"payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.171226 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.171069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e855fb-c3c6-48aa-aab1-6f2a180cb801-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.272328 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.272240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e855fb-c3c6-48aa-aab1-6f2a180cb801-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.272492 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.272376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7dtt\" (UniqueName: \"kubernetes.io/projected/70e855fb-c3c6-48aa-aab1-6f2a180cb801-kube-api-access-d7dtt\") pod \"payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.273133 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.273115    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e855fb-c3c6-48aa-aab1-6f2a180cb801-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.283516 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.283490    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7dtt\" (UniqueName: \"kubernetes.io/projected/70e855fb-c3c6-48aa-aab1-6f2a180cb801-kube-api-access-d7dtt\") pod \"payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.305954 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.305914    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:12.312689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.312665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"]
Sep 10 23:12:12.314710 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:12.314685    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d2a73ba_c225_41c9_add6_a6efc03748bf.slice/crio-6b83fdc16bc94452ae9418b79814d5245642cda73f538b5ce38513b3ac27fb64 WatchSource:0}: Error finding container 6b83fdc16bc94452ae9418b79814d5245642cda73f538b5ce38513b3ac27fb64: Status 404 returned error can't find the container with id 6b83fdc16bc94452ae9418b79814d5245642cda73f538b5ce38513b3ac27fb64
Sep 10 23:12:12.356958 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.356924    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.636109 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.636085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"]
Sep 10 23:12:12.638713 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:12.638681    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70e855fb_c3c6_48aa_aab1_6f2a180cb801.slice/crio-f96ade722845d27150543fd66966b6a1b463b77b8c4249cdf15d517bc16fc466 WatchSource:0}: Error finding container f96ade722845d27150543fd66966b6a1b463b77b8c4249cdf15d517bc16fc466: Status 404 returned error can't find the container with id f96ade722845d27150543fd66966b6a1b463b77b8c4249cdf15d517bc16fc466
Sep 10 23:12:12.766513 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.766480    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws" event={"ID":"9d2a73ba-c225-41c9-add6-a6efc03748bf","Type":"ContainerStarted","Data":"6b83fdc16bc94452ae9418b79814d5245642cda73f538b5ce38513b3ac27fb64"}
Sep 10 23:12:12.768598 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.768566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4" event={"ID":"70e855fb-c3c6-48aa-aab1-6f2a180cb801","Type":"ContainerStarted","Data":"395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b"}
Sep 10 23:12:12.768734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.768618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:12.768734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.768640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4" event={"ID":"70e855fb-c3c6-48aa-aab1-6f2a180cb801","Type":"ContainerStarted","Data":"f96ade722845d27150543fd66966b6a1b463b77b8c4249cdf15d517bc16fc466"}
Sep 10 23:12:12.789895 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:12.789843    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4" podStartSLOduration=0.789826426 podStartE2EDuration="789.826426ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:12.789802429 +0000 UTC m=+849.225857194" watchObservedRunningTime="2026-09-10 23:12:12.789826426 +0000 UTC m=+849.225881199"
Sep 10 23:12:13.775117 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:13.775075    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws" event={"ID":"9d2a73ba-c225-41c9-add6-a6efc03748bf","Type":"ContainerStarted","Data":"a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a"}
Sep 10 23:12:13.798017 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:13.797941    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws" podStartSLOduration=2.30582306 podStartE2EDuration="2.797924379s" totalImagesPullingTime="492.101319ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:13.796079376 +0000 UTC m=+850.232134143" watchObservedRunningTime="2026-09-10 23:12:13.797924379 +0000 UTC m=+850.233979152"
Sep 10 23:12:14.780197 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:14.780163    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:15.610418 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:15.610383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:15.706282 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:15.706235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:16.658655 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.658612    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:16.828845 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.828751    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:16.830157 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.830135    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"]
Sep 10 23:12:16.832902 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.832883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:16.836079 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.836057    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-autores-3a438f-serving-cert\""
Sep 10 23:12:16.853465 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.853437    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"]
Sep 10 23:12:16.911855 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.911760    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w47jf\" (UniqueName: \"kubernetes.io/projected/6a9156d2-3e81-451c-8ac5-317a50da2e3c-kube-api-access-w47jf\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:16.911997 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.911880    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-metrics-certs\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:16.911997 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.911956    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9156d2-3e81-451c-8ac5-317a50da2e3c-openshift-service-ca\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:16.912126 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.912037    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:16.928375 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:16.928343    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:17.012902 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.012861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.013097 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.012935    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w47jf\" (UniqueName: \"kubernetes.io/projected/6a9156d2-3e81-451c-8ac5-317a50da2e3c-kube-api-access-w47jf\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.013097 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.012985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-metrics-certs\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.013097 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:17.013017    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-serving-cert: secret "maas-api-e2e-autores-3a438f-serving-cert" not found
Sep 10 23:12:17.013097 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.013029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9156d2-3e81-451c-8ac5-317a50da2e3c-openshift-service-ca\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.013097 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:17.013082    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls podName:6a9156d2-3e81-451c-8ac5-317a50da2e3c nodeName:}" failed. No retries permitted until 2026-09-10 23:12:17.513062999 +0000 UTC m=+853.949117751 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls") pod "maas-api-e2e-autores-3a438f-596f9d4cd-z86q6" (UID: "6a9156d2-3e81-451c-8ac5-317a50da2e3c") : secret "maas-api-e2e-autores-3a438f-serving-cert" not found
Sep 10 23:12:17.013704 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.013681    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9156d2-3e81-451c-8ac5-317a50da2e3c-openshift-service-ca\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.015676 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.015649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-metrics-certs\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.021583 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.021557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:17.024004 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.023986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w47jf\" (UniqueName: \"kubernetes.io/projected/6a9156d2-3e81-451c-8ac5-317a50da2e3c-kube-api-access-w47jf\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.130847 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.130803    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:17.407460 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.407425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:17.509424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.509386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:17.516660 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.516629    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.519072 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.519053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls\") pod \"maas-api-e2e-autores-3a438f-596f9d4cd-z86q6\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") " pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.611086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.611051    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:17.649343 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.649309    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"]
Sep 10 23:12:17.652391 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.652372    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:17.655157 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.655134    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-3a438f\""
Sep 10 23:12:17.666756 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.666726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"]
Sep 10 23:12:17.718558 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.718515    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9bld\" (UniqueName: \"kubernetes.io/projected/b5524b8f-1944-46fc-9e33-fa3733f2d647-kube-api-access-c9bld\") pod \"payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") " pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:17.718763 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.718699    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5524b8f-1944-46fc-9e33-fa3733f2d647-plugins-config-volume\") pod \"payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") " pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:17.745509 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.745468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:17.821247 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.820165    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9bld\" (UniqueName: \"kubernetes.io/projected/b5524b8f-1944-46fc-9e33-fa3733f2d647-kube-api-access-c9bld\") pod \"payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") " pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:17.821247 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.820294    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5524b8f-1944-46fc-9e33-fa3733f2d647-plugins-config-volume\") pod \"payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") " pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:17.821247 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.821194    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5524b8f-1944-46fc-9e33-fa3733f2d647-plugins-config-volume\") pod \"payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") " pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:17.832941 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.832912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9bld\" (UniqueName: \"kubernetes.io/projected/b5524b8f-1944-46fc-9e33-fa3733f2d647-kube-api-access-c9bld\") pod \"payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") " pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:17.963638 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:17.963545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:18.037963 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.037756    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"]
Sep 10 23:12:18.040275 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:18.040244    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9156d2_3e81_451c_8ac5_317a50da2e3c.slice/crio-5f28db15a90b4118ae53e8679320bb4c8ab39b0cdbc36b399436a1a7578eba11 WatchSource:0}: Error finding container 5f28db15a90b4118ae53e8679320bb4c8ab39b0cdbc36b399436a1a7578eba11: Status 404 returned error can't find the container with id 5f28db15a90b4118ae53e8679320bb4c8ab39b0cdbc36b399436a1a7578eba11
Sep 10 23:12:18.238349 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.238321    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"]
Sep 10 23:12:18.241149 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:18.241121    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5524b8f_1944_46fc_9e33_fa3733f2d647.slice/crio-bd883ea6e9a0acde13af6ebad83d8c0017627e6084a9dc16b91da3578b09c2a0 WatchSource:0}: Error finding container bd883ea6e9a0acde13af6ebad83d8c0017627e6084a9dc16b91da3578b09c2a0: Status 404 returned error can't find the container with id bd883ea6e9a0acde13af6ebad83d8c0017627e6084a9dc16b91da3578b09c2a0
Sep 10 23:12:18.803259 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.803147    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6" event={"ID":"6a9156d2-3e81-451c-8ac5-317a50da2e3c","Type":"ContainerStarted","Data":"b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8"}
Sep 10 23:12:18.803689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.803310    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:18.803689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.803355    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6" event={"ID":"6a9156d2-3e81-451c-8ac5-317a50da2e3c","Type":"ContainerStarted","Data":"5f28db15a90b4118ae53e8679320bb4c8ab39b0cdbc36b399436a1a7578eba11"}
Sep 10 23:12:18.804898 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.804875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw" event={"ID":"b5524b8f-1944-46fc-9e33-fa3733f2d647","Type":"ContainerStarted","Data":"7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a"}
Sep 10 23:12:18.805005 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.804913    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw" event={"ID":"b5524b8f-1944-46fc-9e33-fa3733f2d647","Type":"ContainerStarted","Data":"bd883ea6e9a0acde13af6ebad83d8c0017627e6084a9dc16b91da3578b09c2a0"}
Sep 10 23:12:18.805005 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.804937    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:18.826152 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.826103    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6" podStartSLOduration=2.375284506 podStartE2EDuration="2.826088926s" totalImagesPullingTime="450.80442ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:18.823530767 +0000 UTC m=+855.259585543" watchObservedRunningTime="2026-09-10 23:12:18.826088926 +0000 UTC m=+855.262143750"
Sep 10 23:12:18.844832 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:18.844768    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw" podStartSLOduration=1.844753314 podStartE2EDuration="1.844753314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:18.843485311 +0000 UTC m=+855.279540084" watchObservedRunningTime="2026-09-10 23:12:18.844753314 +0000 UTC m=+855.280808087"
Sep 10 23:12:19.716263 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:19.716215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:20.791111 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:20.791077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:20.916791 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:20.916730    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:21.012731 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:21.012686    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:21.133649 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:21.133544    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:21.338431 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:21.338393    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:23.777318 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:23.777288    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:24.019869 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:24.019836    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:24.814424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:24.814396    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:12:26.429654 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:26.429616    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:26.764825 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:26.764728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:27.070586 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:27.070491    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:27.694247 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:27.694209    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:28.609667 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:28.609633    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:28.733323 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:28.733278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:28.814265 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:28.814228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:29.045265 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:29.045235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:29.812105 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:29.812077    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:12:30.775809 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:30.775741    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:30.875498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:30.875457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:32.076849 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.076816    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"]
Sep 10 23:12:32.079621 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.079598    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.083270 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.083252    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-1f1f85-serving-cert\""
Sep 10 23:12:32.096554 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.096531    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"]
Sep 10 23:12:32.137232 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.137195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:32.151318 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.151282    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stx5\" (UniqueName: \"kubernetes.io/projected/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-kube-api-access-8stx5\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.151456 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.151336    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.151456 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.151400    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.151530 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.151463    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-metrics-certs\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.238321 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.238278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:32.252861 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.252831    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stx5\" (UniqueName: \"kubernetes.io/projected/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-kube-api-access-8stx5\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.253049 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.252870    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.253049 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.252898    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.253049 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.252932    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-metrics-certs\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.253049 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:32.253016    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-serving-cert: secret "maas-api-e2e-inf-b-1f1f85-serving-cert" not found
Sep 10 23:12:32.253260 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:32.253086    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls podName:2891bfb3-de47-4c10-8c39-2b1d1be4e0a0 nodeName:}" failed. No retries permitted until 2026-09-10 23:12:32.753069294 +0000 UTC m=+869.189124051 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls") pod "maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x" (UID: "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0") : secret "maas-api-e2e-inf-b-1f1f85-serving-cert" not found
Sep 10 23:12:32.253615 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.253593    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-openshift-service-ca\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.255550 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.255530    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-metrics-certs\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.264696 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.264662    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stx5\" (UniqueName: \"kubernetes.io/projected/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-kube-api-access-8stx5\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.408026 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.407996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:32.758638 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.758540    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.761470 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.761416    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls\") pod \"maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:32.813446 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.813401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:32.913615 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.913580    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:32.991540 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:32.991503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:33.115300 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.115261    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"]
Sep 10 23:12:33.117988 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.117965    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.121567 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.121545    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-1f1f85\""
Sep 10 23:12:33.121677 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.121581    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-1f1f85-dockercfg-jwjbx\""
Sep 10 23:12:33.133467 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.133439    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"]
Sep 10 23:12:33.162956 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.162924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc155a6-61a5-4113-ae8f-b080310aea50-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.163110 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.162970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64x95\" (UniqueName: \"kubernetes.io/projected/8fc155a6-61a5-4113-ae8f-b080310aea50-kube-api-access-64x95\") pod \"payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.264265 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.264216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc155a6-61a5-4113-ae8f-b080310aea50-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.264451 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.264376    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64x95\" (UniqueName: \"kubernetes.io/projected/8fc155a6-61a5-4113-ae8f-b080310aea50-kube-api-access-64x95\") pod \"payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.265191 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.265168    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc155a6-61a5-4113-ae8f-b080310aea50-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.265539 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.265519    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"]
Sep 10 23:12:33.268212 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:33.268183    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2891bfb3_de47_4c10_8c39_2b1d1be4e0a0.slice/crio-55a48b3cdf74d3377acdeec864dc4a8c32c68cd76d2e90b72fb53261c0cdbf1e WatchSource:0}: Error finding container 55a48b3cdf74d3377acdeec864dc4a8c32c68cd76d2e90b72fb53261c0cdbf1e: Status 404 returned error can't find the container with id 55a48b3cdf74d3377acdeec864dc4a8c32c68cd76d2e90b72fb53261c0cdbf1e
Sep 10 23:12:33.274505 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.274482    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-64x95\" (UniqueName: \"kubernetes.io/projected/8fc155a6-61a5-4113-ae8f-b080310aea50-kube-api-access-64x95\") pod \"payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") " pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.428995 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.428952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:33.431606 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.431587    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.719558 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.719522    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"]
Sep 10 23:12:33.724107 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:33.724080    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fc155a6_61a5_4113_ae8f_b080310aea50.slice/crio-15d35ab56acaae92a23a13d8ecf30055131f40f41d1719595e1593c43fc6af1d WatchSource:0}: Error finding container 15d35ab56acaae92a23a13d8ecf30055131f40f41d1719595e1593c43fc6af1d: Status 404 returned error can't find the container with id 15d35ab56acaae92a23a13d8ecf30055131f40f41d1719595e1593c43fc6af1d
Sep 10 23:12:33.880903 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.880871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp" event={"ID":"8fc155a6-61a5-4113-ae8f-b080310aea50","Type":"ContainerStarted","Data":"ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd"}
Sep 10 23:12:33.881066 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.880922    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp" event={"ID":"8fc155a6-61a5-4113-ae8f-b080310aea50","Type":"ContainerStarted","Data":"15d35ab56acaae92a23a13d8ecf30055131f40f41d1719595e1593c43fc6af1d"}
Sep 10 23:12:33.881066 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.880978    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:33.882664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.882640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x" event={"ID":"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0","Type":"ContainerStarted","Data":"4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8"}
Sep 10 23:12:33.882770 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.882678    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x" event={"ID":"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0","Type":"ContainerStarted","Data":"55a48b3cdf74d3377acdeec864dc4a8c32c68cd76d2e90b72fb53261c0cdbf1e"}
Sep 10 23:12:33.882770 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.882734    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:33.921384 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.921337    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp" podStartSLOduration=0.921321855 podStartE2EDuration="921.321855ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:33.918539873 +0000 UTC m=+870.354594645" watchObservedRunningTime="2026-09-10 23:12:33.921321855 +0000 UTC m=+870.357376627"
Sep 10 23:12:33.950022 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:33.949929    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x" podStartSLOduration=1.606798011 podStartE2EDuration="1.949914045s" totalImagesPullingTime="343.116034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:33.949126909 +0000 UTC m=+870.385181683" watchObservedRunningTime="2026-09-10 23:12:33.949914045 +0000 UTC m=+870.385968818"
Sep 10 23:12:34.622132 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:34.622098    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:35.555632 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:35.555588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:37.091338 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:37.091301    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:37.188129 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:37.188093    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:38.050200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:38.050158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:39.893510 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:39.893480    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:12:40.542015 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:40.541976    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:44.890370 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:44.890339    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:12:51.441031 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:51.440989    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:52.216193 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.216157    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"]
Sep 10 23:12:52.216476 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.216416    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws" podUID="9d2a73ba-c225-41c9-add6-a6efc03748bf" containerName="maas-api" containerID="cri-o://a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a" gracePeriod=30
Sep 10 23:12:52.240585 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.240550    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:52.303453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.303412    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"]
Sep 10 23:12:52.303755 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.303718    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4" podUID="70e855fb-c3c6-48aa-aab1-6f2a180cb801" containerName="payload-pre-processing" containerID="cri-o://395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b" gracePeriod=30
Sep 10 23:12:52.334480 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.334439    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:52.408448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.408389    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:52.503091 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.503053    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:52.542671 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.542641    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:52.572817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.572733    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:52.575610 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.575589    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:52.653563 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.653534    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2a73ba-c225-41c9-add6-a6efc03748bf-openshift-service-ca\") pod \"9d2a73ba-c225-41c9-add6-a6efc03748bf\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") "
Sep 10 23:12:52.653746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.653579    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-metrics-certs\") pod \"9d2a73ba-c225-41c9-add6-a6efc03748bf\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") "
Sep 10 23:12:52.653746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.653626    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls\") pod \"9d2a73ba-c225-41c9-add6-a6efc03748bf\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") "
Sep 10 23:12:52.653746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.653654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9sqsw\" (UniqueName: \"kubernetes.io/projected/9d2a73ba-c225-41c9-add6-a6efc03748bf-kube-api-access-9sqsw\") pod \"9d2a73ba-c225-41c9-add6-a6efc03748bf\" (UID: \"9d2a73ba-c225-41c9-add6-a6efc03748bf\") "
Sep 10 23:12:52.653990 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.653951    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d2a73ba-c225-41c9-add6-a6efc03748bf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9d2a73ba-c225-41c9-add6-a6efc03748bf" (UID: "9d2a73ba-c225-41c9-add6-a6efc03748bf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:12:52.655053 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.655023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:52.656044 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.656019    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9d2a73ba-c225-41c9-add6-a6efc03748bf" (UID: "9d2a73ba-c225-41c9-add6-a6efc03748bf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:12:52.656151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.656068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9d2a73ba-c225-41c9-add6-a6efc03748bf" (UID: "9d2a73ba-c225-41c9-add6-a6efc03748bf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:12:52.656151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.656110    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d2a73ba-c225-41c9-add6-a6efc03748bf-kube-api-access-9sqsw" (OuterVolumeSpecName: "kube-api-access-9sqsw") pod "9d2a73ba-c225-41c9-add6-a6efc03748bf" (UID: "9d2a73ba-c225-41c9-add6-a6efc03748bf"). InnerVolumeSpecName "kube-api-access-9sqsw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:12:52.754835 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.754752    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7dtt\" (UniqueName: \"kubernetes.io/projected/70e855fb-c3c6-48aa-aab1-6f2a180cb801-kube-api-access-d7dtt\") pod \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") "
Sep 10 23:12:52.754835 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.754824    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e855fb-c3c6-48aa-aab1-6f2a180cb801-plugins-config-volume\") pod \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\" (UID: \"70e855fb-c3c6-48aa-aab1-6f2a180cb801\") "
Sep 10 23:12:52.755128 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.755107    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d2a73ba-c225-41c9-add6-a6efc03748bf-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:12:52.755171 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.755134    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:12:52.755171 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.755148    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9d2a73ba-c225-41c9-add6-a6efc03748bf-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:12:52.755234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.755161    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sqsw\" (UniqueName: \"kubernetes.io/projected/9d2a73ba-c225-41c9-add6-a6efc03748bf-kube-api-access-9sqsw\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:12:52.755234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.755182    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70e855fb-c3c6-48aa-aab1-6f2a180cb801-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70e855fb-c3c6-48aa-aab1-6f2a180cb801" (UID: "70e855fb-c3c6-48aa-aab1-6f2a180cb801"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:12:52.756985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.756963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70e855fb-c3c6-48aa-aab1-6f2a180cb801-kube-api-access-d7dtt" (OuterVolumeSpecName: "kube-api-access-d7dtt") pod "70e855fb-c3c6-48aa-aab1-6f2a180cb801" (UID: "70e855fb-c3c6-48aa-aab1-6f2a180cb801"). InnerVolumeSpecName "kube-api-access-d7dtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:12:52.856148 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.856111    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7dtt\" (UniqueName: \"kubernetes.io/projected/70e855fb-c3c6-48aa-aab1-6f2a180cb801-kube-api-access-d7dtt\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:12:52.856148 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.856142    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70e855fb-c3c6-48aa-aab1-6f2a180cb801-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:12:52.976482 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.976363    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d2a73ba-c225-41c9-add6-a6efc03748bf" containerID="a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a" exitCode=0
Sep 10 23:12:52.976482 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.976445    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"
Sep 10 23:12:52.976482 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.976479    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws" event={"ID":"9d2a73ba-c225-41c9-add6-a6efc03748bf","Type":"ContainerDied","Data":"a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a"}
Sep 10 23:12:52.976825 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.976542    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws" event={"ID":"9d2a73ba-c225-41c9-add6-a6efc03748bf","Type":"ContainerDied","Data":"6b83fdc16bc94452ae9418b79814d5245642cda73f538b5ce38513b3ac27fb64"}
Sep 10 23:12:52.976825 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.976573    2577 scope.go:117] "RemoveContainer" containerID="a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a"
Sep 10 23:12:52.977993 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.977968    2577 generic.go:356] "Generic (PLEG): container finished" podID="70e855fb-c3c6-48aa-aab1-6f2a180cb801" containerID="395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b" exitCode=0
Sep 10 23:12:52.978115 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.978017    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"
Sep 10 23:12:52.978115 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.978034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4" event={"ID":"70e855fb-c3c6-48aa-aab1-6f2a180cb801","Type":"ContainerDied","Data":"395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b"}
Sep 10 23:12:52.978115 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.978069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4" event={"ID":"70e855fb-c3c6-48aa-aab1-6f2a180cb801","Type":"ContainerDied","Data":"f96ade722845d27150543fd66966b6a1b463b77b8c4249cdf15d517bc16fc466"}
Sep 10 23:12:52.987162 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.987122    2577 scope.go:117] "RemoveContainer" containerID="a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a"
Sep 10 23:12:52.987532 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:52.987507    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a\": container with ID starting with a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a not found: ID does not exist" containerID="a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a"
Sep 10 23:12:52.987644 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.987542    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a"} err="failed to get container status \"a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a\": rpc error: code = NotFound desc = could not find container \"a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a\": container with ID starting with a1611a88045312e71e30e4ae2b5e8e9cff4e3cd52e5ae2af4189aef34b57024a not found: ID does not exist"
Sep 10 23:12:52.987644 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.987566    2577 scope.go:117] "RemoveContainer" containerID="395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b"
Sep 10 23:12:52.996839 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.996821    2577 scope.go:117] "RemoveContainer" containerID="395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b"
Sep 10 23:12:52.997102 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:52.997081    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b\": container with ID starting with 395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b not found: ID does not exist" containerID="395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b"
Sep 10 23:12:52.997170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:52.997107    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b"} err="failed to get container status \"395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b\": rpc error: code = NotFound desc = could not find container \"395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b\": container with ID starting with 395aa96b4a5f93f50d67ed3362042d1c6f3ae3a6f54de52df672ec2eda12128b not found: ID does not exist"
Sep 10 23:12:53.005041 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.004990    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"]
Sep 10 23:12:53.010554 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.010534    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-e7db0e72-fcf7b8ccd-pmsk4"]
Sep 10 23:12:53.023998 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.023975    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"]
Sep 10 23:12:53.031562 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.031541    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-e7db0e72-7c7cbb8746-r6lws"]
Sep 10 23:12:53.111165 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.111127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:53.207967 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.207929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:53.310706 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.310618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:53.393092 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.393061    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:53.622979 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.618555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:53.756028 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:53.755984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:54.215267 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:54.215229    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70e855fb-c3c6-48aa-aab1-6f2a180cb801" path="/var/lib/kubelet/pods/70e855fb-c3c6-48aa-aab1-6f2a180cb801/volumes"
Sep 10 23:12:54.215645 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:54.215632    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9d2a73ba-c225-41c9-add6-a6efc03748bf" path="/var/lib/kubelet/pods/9d2a73ba-c225-41c9-add6-a6efc03748bf/volumes"
Sep 10 23:12:55.691271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:55.691230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:56.440094 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.440055    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:56.543831 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.543776    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:56.587973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.587939    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"]
Sep 10 23:12:56.588412 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.588388    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d2a73ba-c225-41c9-add6-a6efc03748bf" containerName="maas-api"
Sep 10 23:12:56.588412 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.588410    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d2a73ba-c225-41c9-add6-a6efc03748bf" containerName="maas-api"
Sep 10 23:12:56.588630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.588440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70e855fb-c3c6-48aa-aab1-6f2a180cb801" containerName="payload-pre-processing"
Sep 10 23:12:56.588630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.588450    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="70e855fb-c3c6-48aa-aab1-6f2a180cb801" containerName="payload-pre-processing"
Sep 10 23:12:56.588630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.588538    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="70e855fb-c3c6-48aa-aab1-6f2a180cb801" containerName="payload-pre-processing"
Sep 10 23:12:56.588630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.588550    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d2a73ba-c225-41c9-add6-a6efc03748bf" containerName="maas-api"
Sep 10 23:12:56.592108 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.592085    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.594899 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.594873    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-4a0c9f43-serving-cert\""
Sep 10 23:12:56.605033 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.605007    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"]
Sep 10 23:12:56.633973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.633939    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:56.694206 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.694106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da050c59-688f-41e6-9081-b7ea8bf62ca2-openshift-service-ca\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.694206 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.694158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.694824 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.694230    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n8s7\" (UniqueName: \"kubernetes.io/projected/da050c59-688f-41e6-9081-b7ea8bf62ca2-kube-api-access-6n8s7\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.694824 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.694300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-metrics-certs\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.694998 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.694963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:56.794898 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.794854    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-metrics-certs\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.795086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.794955    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da050c59-688f-41e6-9081-b7ea8bf62ca2-openshift-service-ca\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.795086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.794989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.795086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.795029    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n8s7\" (UniqueName: \"kubernetes.io/projected/da050c59-688f-41e6-9081-b7ea8bf62ca2-kube-api-access-6n8s7\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.795318 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:56.795136    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-serving-cert: secret "maas-api-e2e-migrate-4a0c9f43-serving-cert" not found
Sep 10 23:12:56.795318 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:12:56.795221    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls podName:da050c59-688f-41e6-9081-b7ea8bf62ca2 nodeName:}" failed. No retries permitted until 2026-09-10 23:12:57.295201862 +0000 UTC m=+893.731256613 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls") pod "maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw" (UID: "da050c59-688f-41e6-9081-b7ea8bf62ca2") : secret "maas-api-e2e-migrate-4a0c9f43-serving-cert" not found
Sep 10 23:12:56.795690 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.795671    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da050c59-688f-41e6-9081-b7ea8bf62ca2-openshift-service-ca\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.797625 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.797604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-metrics-certs\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.804262 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.804235    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n8s7\" (UniqueName: \"kubernetes.io/projected/da050c59-688f-41e6-9081-b7ea8bf62ca2-kube-api-access-6n8s7\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:56.996616 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:56.996507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:57.223227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.223195    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:57.298529 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.298440    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:57.300775 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.300750    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls\") pod \"maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:57.324935 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.324903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:12:57.331381 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.331355    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"]
Sep 10 23:12:57.335001 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.334980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.337220 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.337200    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-4a0c9f43\""
Sep 10 23:12:57.346658 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.346637    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"]
Sep 10 23:12:57.399375 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.399337    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111e4dd3-ef03-495e-a09a-e20d328626aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.399542 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.399382    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/111e4dd3-ef03-495e-a09a-e20d328626aa-kube-api-access-qdw4f\") pod \"payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.500327 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.500289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111e4dd3-ef03-495e-a09a-e20d328626aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.500499 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.500343    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/111e4dd3-ef03-495e-a09a-e20d328626aa-kube-api-access-qdw4f\") pod \"payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.501161 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.501141    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111e4dd3-ef03-495e-a09a-e20d328626aa-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.503198 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.503178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:57.509726 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.509668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/111e4dd3-ef03-495e-a09a-e20d328626aa-kube-api-access-qdw4f\") pod \"payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.646396 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.646342    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:57.780655 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.780631    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"]
Sep 10 23:12:57.782583 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:57.782558    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda050c59_688f_41e6_9081_b7ea8bf62ca2.slice/crio-7ccac3b6d58d1ca2d4cef81e8f3bc3132b5958dce5e6b9ec7f68fc06d7985fac WatchSource:0}: Error finding container 7ccac3b6d58d1ca2d4cef81e8f3bc3132b5958dce5e6b9ec7f68fc06d7985fac: Status 404 returned error can't find the container with id 7ccac3b6d58d1ca2d4cef81e8f3bc3132b5958dce5e6b9ec7f68fc06d7985fac
Sep 10 23:12:57.918740 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:57.918688    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"]
Sep 10 23:12:57.921405 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:12:57.921381    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod111e4dd3_ef03_495e_a09a_e20d328626aa.slice/crio-406b329baff517cadd6b74580aa6d838890972f6e3d1dc43f121009623cf7df6 WatchSource:0}: Error finding container 406b329baff517cadd6b74580aa6d838890972f6e3d1dc43f121009623cf7df6: Status 404 returned error can't find the container with id 406b329baff517cadd6b74580aa6d838890972f6e3d1dc43f121009623cf7df6
Sep 10 23:12:58.006828 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:58.006775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw" event={"ID":"da050c59-688f-41e6-9081-b7ea8bf62ca2","Type":"ContainerStarted","Data":"7ccac3b6d58d1ca2d4cef81e8f3bc3132b5958dce5e6b9ec7f68fc06d7985fac"}
Sep 10 23:12:58.008470 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:58.008446    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q" event={"ID":"111e4dd3-ef03-495e-a09a-e20d328626aa","Type":"ContainerStarted","Data":"fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242"}
Sep 10 23:12:58.008603 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:58.008493    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q" event={"ID":"111e4dd3-ef03-495e-a09a-e20d328626aa","Type":"ContainerStarted","Data":"406b329baff517cadd6b74580aa6d838890972f6e3d1dc43f121009623cf7df6"}
Sep 10 23:12:58.008603 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:58.008569    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:12:58.026572 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:58.026522    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q" podStartSLOduration=1.026505592 podStartE2EDuration="1.026505592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:58.024081027 +0000 UTC m=+894.460135788" watchObservedRunningTime="2026-09-10 23:12:58.026505592 +0000 UTC m=+894.462560356"
Sep 10 23:12:59.016216 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:59.016174    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw" event={"ID":"da050c59-688f-41e6-9081-b7ea8bf62ca2","Type":"ContainerStarted","Data":"2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f"}
Sep 10 23:12:59.016618 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:59.016375    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:12:59.036283 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:12:59.036226    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw" podStartSLOduration=2.507509861 podStartE2EDuration="3.03620939s" totalImagesPullingTime="528.699529ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:12:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:12:59.034341901 +0000 UTC m=+895.470396704" watchObservedRunningTime="2026-09-10 23:12:59.03620939 +0000 UTC m=+895.472264163"
Sep 10 23:13:05.026944 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:05.026913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:13:09.018972 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:09.018930    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:13:10.576073 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:10.576034    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:28.108719 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.108581    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"]
Sep 10 23:13:28.111631 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.108852    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw" podUID="da050c59-688f-41e6-9081-b7ea8bf62ca2" containerName="maas-api" containerID="cri-o://2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f" gracePeriod=30
Sep 10 23:13:28.128065 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.128032    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:28.166504 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.166469    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"]
Sep 10 23:13:28.166701 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.166677    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q" podUID="111e4dd3-ef03-495e-a09a-e20d328626aa" containerName="payload-pre-processing" containerID="cri-o://fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242" gracePeriod=30
Sep 10 23:13:28.199918 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.199887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:28.257437 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.257386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:28.357684 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.357648    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:28.410539 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.410512    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:13:28.440654 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.440597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:28.447068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.447045    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:13:28.484346 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.484313    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-metrics-certs\") pod \"da050c59-688f-41e6-9081-b7ea8bf62ca2\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") "
Sep 10 23:13:28.484536 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.484371    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da050c59-688f-41e6-9081-b7ea8bf62ca2-openshift-service-ca\") pod \"da050c59-688f-41e6-9081-b7ea8bf62ca2\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") "
Sep 10 23:13:28.484536 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.484402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls\") pod \"da050c59-688f-41e6-9081-b7ea8bf62ca2\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") "
Sep 10 23:13:28.484536 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.484428    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6n8s7\" (UniqueName: \"kubernetes.io/projected/da050c59-688f-41e6-9081-b7ea8bf62ca2-kube-api-access-6n8s7\") pod \"da050c59-688f-41e6-9081-b7ea8bf62ca2\" (UID: \"da050c59-688f-41e6-9081-b7ea8bf62ca2\") "
Sep 10 23:13:28.484886 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.484832    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da050c59-688f-41e6-9081-b7ea8bf62ca2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "da050c59-688f-41e6-9081-b7ea8bf62ca2" (UID: "da050c59-688f-41e6-9081-b7ea8bf62ca2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:13:28.487148 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.487124    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da050c59-688f-41e6-9081-b7ea8bf62ca2-kube-api-access-6n8s7" (OuterVolumeSpecName: "kube-api-access-6n8s7") pod "da050c59-688f-41e6-9081-b7ea8bf62ca2" (UID: "da050c59-688f-41e6-9081-b7ea8bf62ca2"). InnerVolumeSpecName "kube-api-access-6n8s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:13:28.488880 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.488849    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "da050c59-688f-41e6-9081-b7ea8bf62ca2" (UID: "da050c59-688f-41e6-9081-b7ea8bf62ca2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:13:28.488982 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.488919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "da050c59-688f-41e6-9081-b7ea8bf62ca2" (UID: "da050c59-688f-41e6-9081-b7ea8bf62ca2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:13:28.542232 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.542197    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:28.586084 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.586051    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/111e4dd3-ef03-495e-a09a-e20d328626aa-kube-api-access-qdw4f\") pod \"111e4dd3-ef03-495e-a09a-e20d328626aa\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") "
Sep 10 23:13:28.586259 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.586106    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111e4dd3-ef03-495e-a09a-e20d328626aa-plugins-config-volume\") pod \"111e4dd3-ef03-495e-a09a-e20d328626aa\" (UID: \"111e4dd3-ef03-495e-a09a-e20d328626aa\") "
Sep 10 23:13:28.586306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.586292    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:13:28.586341 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.586311    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/da050c59-688f-41e6-9081-b7ea8bf62ca2-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:13:28.586341 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.586326    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/da050c59-688f-41e6-9081-b7ea8bf62ca2-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:13:28.586341 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.586338    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n8s7\" (UniqueName: \"kubernetes.io/projected/da050c59-688f-41e6-9081-b7ea8bf62ca2-kube-api-access-6n8s7\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:13:28.586487 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.586470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/111e4dd3-ef03-495e-a09a-e20d328626aa-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "111e4dd3-ef03-495e-a09a-e20d328626aa" (UID: "111e4dd3-ef03-495e-a09a-e20d328626aa"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:13:28.588176 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.588152    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/111e4dd3-ef03-495e-a09a-e20d328626aa-kube-api-access-qdw4f" (OuterVolumeSpecName: "kube-api-access-qdw4f") pod "111e4dd3-ef03-495e-a09a-e20d328626aa" (UID: "111e4dd3-ef03-495e-a09a-e20d328626aa"). InnerVolumeSpecName "kube-api-access-qdw4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:13:28.687239 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.687206    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdw4f\" (UniqueName: \"kubernetes.io/projected/111e4dd3-ef03-495e-a09a-e20d328626aa-kube-api-access-qdw4f\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:13:28.687239 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.687233    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/111e4dd3-ef03-495e-a09a-e20d328626aa-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:13:28.965099 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:28.965007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:29.175465 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.175426    2577 generic.go:356] "Generic (PLEG): container finished" podID="111e4dd3-ef03-495e-a09a-e20d328626aa" containerID="fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242" exitCode=0
Sep 10 23:13:29.175955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.175489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"
Sep 10 23:13:29.175955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.175509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q" event={"ID":"111e4dd3-ef03-495e-a09a-e20d328626aa","Type":"ContainerDied","Data":"fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242"}
Sep 10 23:13:29.175955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.175605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q" event={"ID":"111e4dd3-ef03-495e-a09a-e20d328626aa","Type":"ContainerDied","Data":"406b329baff517cadd6b74580aa6d838890972f6e3d1dc43f121009623cf7df6"}
Sep 10 23:13:29.175955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.175637    2577 scope.go:117] "RemoveContainer" containerID="fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242"
Sep 10 23:13:29.177368 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.177275    2577 generic.go:356] "Generic (PLEG): container finished" podID="da050c59-688f-41e6-9081-b7ea8bf62ca2" containerID="2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f" exitCode=0
Sep 10 23:13:29.177368 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.177341    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"
Sep 10 23:13:29.177473 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.177361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw" event={"ID":"da050c59-688f-41e6-9081-b7ea8bf62ca2","Type":"ContainerDied","Data":"2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f"}
Sep 10 23:13:29.177473 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.177429    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw" event={"ID":"da050c59-688f-41e6-9081-b7ea8bf62ca2","Type":"ContainerDied","Data":"7ccac3b6d58d1ca2d4cef81e8f3bc3132b5958dce5e6b9ec7f68fc06d7985fac"}
Sep 10 23:13:29.185282 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.185241    2577 scope.go:117] "RemoveContainer" containerID="fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242"
Sep 10 23:13:29.185639 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:13:29.185587    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242\": container with ID starting with fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242 not found: ID does not exist" containerID="fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242"
Sep 10 23:13:29.185755 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.185645    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242"} err="failed to get container status \"fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242\": rpc error: code = NotFound desc = could not find container \"fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242\": container with ID starting with fb8a50bc40bbc50a15222f4d1fc275b226076872330715822b7b51420e8fc242 not found: ID does not exist"
Sep 10 23:13:29.185755 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.185708    2577 scope.go:117] "RemoveContainer" containerID="2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f"
Sep 10 23:13:29.194336 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.194236    2577 scope.go:117] "RemoveContainer" containerID="2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f"
Sep 10 23:13:29.194512 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:13:29.194493    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f\": container with ID starting with 2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f not found: ID does not exist" containerID="2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f"
Sep 10 23:13:29.194558 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.194519    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f"} err="failed to get container status \"2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f\": rpc error: code = NotFound desc = could not find container \"2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f\": container with ID starting with 2834907c00d0ff28c958d5028f213f8b159ccc0b65e925552fd0a30e6110164f not found: ID does not exist"
Sep 10 23:13:29.201344 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.201322    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"]
Sep 10 23:13:29.211380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.211354    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-4a0c9f43-64b76957fd-dww6q"]
Sep 10 23:13:29.223192 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.223126    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"]
Sep 10 23:13:29.232706 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.232664    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-4a0c9f43-757f9457b7-7jzzw"]
Sep 10 23:13:29.281466 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.281422    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:29.404137 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:29.404103    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:30.214645 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:30.214610    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="111e4dd3-ef03-495e-a09a-e20d328626aa" path="/var/lib/kubelet/pods/111e4dd3-ef03-495e-a09a-e20d328626aa/volumes"
Sep 10 23:13:30.215037 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:30.215028    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da050c59-688f-41e6-9081-b7ea8bf62ca2" path="/var/lib/kubelet/pods/da050c59-688f-41e6-9081-b7ea8bf62ca2/volumes"
Sep 10 23:13:31.762979 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:31.762931    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:31.858583 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:31.858548    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:32.943225 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.943193    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"]
Sep 10 23:13:32.943762 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.943747    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da050c59-688f-41e6-9081-b7ea8bf62ca2" containerName="maas-api"
Sep 10 23:13:32.943829 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.943765    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="da050c59-688f-41e6-9081-b7ea8bf62ca2" containerName="maas-api"
Sep 10 23:13:32.943829 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.943803    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="111e4dd3-ef03-495e-a09a-e20d328626aa" containerName="payload-pre-processing"
Sep 10 23:13:32.943829 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.943813    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="111e4dd3-ef03-495e-a09a-e20d328626aa" containerName="payload-pre-processing"
Sep 10 23:13:32.943926 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.943909    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="111e4dd3-ef03-495e-a09a-e20d328626aa" containerName="payload-pre-processing"
Sep 10 23:13:32.943926 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.943923    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="da050c59-688f-41e6-9081-b7ea8bf62ca2" containerName="maas-api"
Sep 10 23:13:32.947694 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.947669    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:32.950231 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.950210    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-00575c78-serving-cert\""
Sep 10 23:13:32.956414 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.956392    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"]
Sep 10 23:13:32.979384 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:32.979355    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:33.027969 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.027936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-openshift-service-ca\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.028156 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.027994    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.028156 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.028012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-metrics-certs\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.028156 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.028039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6rct\" (UniqueName: \"kubernetes.io/projected/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-kube-api-access-w6rct\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.072296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.072260    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:33.128831 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.128789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.129019 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.128839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-metrics-certs\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.129019 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.128900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6rct\" (UniqueName: \"kubernetes.io/projected/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-kube-api-access-w6rct\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.129019 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:13:33.128919    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-serving-cert: secret "maas-api-e2e-ait-00575c78-serving-cert" not found
Sep 10 23:13:33.129019 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:13:33.128989    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls podName:7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2 nodeName:}" failed. No retries permitted until 2026-09-10 23:13:33.628969543 +0000 UTC m=+930.065024298 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls") pod "maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f" (UID: "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2") : secret "maas-api-e2e-ait-00575c78-serving-cert" not found
Sep 10 23:13:33.129019 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.129011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-openshift-service-ca\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.129664 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.129643    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-openshift-service-ca\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.131281 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.131260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-metrics-certs\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.143143 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.143109    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6rct\" (UniqueName: \"kubernetes.io/projected/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-kube-api-access-w6rct\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.491761 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.491714    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:33.634095 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.634050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.636760 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.636733    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls\") pod \"maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.649859 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.649815    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:33.749854 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.749707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:33.779340 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.779301    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"]
Sep 10 23:13:33.782998 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.782975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:33.785364 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.785342    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-00575c78-dockercfg-hp7d9\""
Sep 10 23:13:33.785453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.785381    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-00575c78\""
Sep 10 23:13:33.793849 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.793815    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"]
Sep 10 23:13:33.861936 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.861903    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:33.939415 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.939383    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2g95\" (UniqueName: \"kubernetes.io/projected/83522d07-0473-46be-8257-333f664b33d0-kube-api-access-g2g95\") pod \"payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:33.939545 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:33.939528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83522d07-0473-46be-8257-333f664b33d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:34.040192 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.040106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83522d07-0473-46be-8257-333f664b33d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:34.040192 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.040153    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2g95\" (UniqueName: \"kubernetes.io/projected/83522d07-0473-46be-8257-333f664b33d0-kube-api-access-g2g95\") pod \"payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:34.040958 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.040936    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83522d07-0473-46be-8257-333f664b33d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:34.048879 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.048858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2g95\" (UniqueName: \"kubernetes.io/projected/83522d07-0473-46be-8257-333f664b33d0-kube-api-access-g2g95\") pod \"payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:34.095773 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.095742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:34.152436 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.152394    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"]
Sep 10 23:13:34.156264 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:13:34.156144    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a1b4fe9_f127_4c4e_acfd_37e8f42a6dd2.slice/crio-d65b353d98701e25cd00ee0581d77cb62190f2f181f119e97609b77a414b5e23 WatchSource:0}: Error finding container d65b353d98701e25cd00ee0581d77cb62190f2f181f119e97609b77a414b5e23: Status 404 returned error can't find the container with id d65b353d98701e25cd00ee0581d77cb62190f2f181f119e97609b77a414b5e23
Sep 10 23:13:34.205810 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.205749    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f" event={"ID":"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2","Type":"ContainerStarted","Data":"d65b353d98701e25cd00ee0581d77cb62190f2f181f119e97609b77a414b5e23"}
Sep 10 23:13:34.372731 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:34.372683    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"]
Sep 10 23:13:34.375707 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:13:34.375679    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83522d07_0473_46be_8257_333f664b33d0.slice/crio-289c270ed417049aa235d7a038d85c5786700e3790a48ede8fed3c3eb63ff011 WatchSource:0}: Error finding container 289c270ed417049aa235d7a038d85c5786700e3790a48ede8fed3c3eb63ff011: Status 404 returned error can't find the container with id 289c270ed417049aa235d7a038d85c5786700e3790a48ede8fed3c3eb63ff011
Sep 10 23:13:35.212099 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.212062    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f" event={"ID":"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2","Type":"ContainerStarted","Data":"4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad"}
Sep 10 23:13:35.212604 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.212155    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:35.213843 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.213803    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc" event={"ID":"83522d07-0473-46be-8257-333f664b33d0","Type":"ContainerStarted","Data":"7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece"}
Sep 10 23:13:35.213964 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.213865    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:35.213964 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.213882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc" event={"ID":"83522d07-0473-46be-8257-333f664b33d0","Type":"ContainerStarted","Data":"289c270ed417049aa235d7a038d85c5786700e3790a48ede8fed3c3eb63ff011"}
Sep 10 23:13:35.237529 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.237471    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f" podStartSLOduration=2.808304034 podStartE2EDuration="3.237454747s" totalImagesPullingTime="429.150713ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:13:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:13:35.236751411 +0000 UTC m=+931.672806184" watchObservedRunningTime="2026-09-10 23:13:35.237454747 +0000 UTC m=+931.673509519"
Sep 10 23:13:35.256409 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.256354    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc" podStartSLOduration=2.256334955 podStartE2EDuration="2.256334955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:13:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:13:35.256071356 +0000 UTC m=+931.692126128" watchObservedRunningTime="2026-09-10 23:13:35.256334955 +0000 UTC m=+931.692389729"
Sep 10 23:13:35.905725 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:35.905685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:41.223347 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:41.223314    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:13:46.221524 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:46.221492    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:13:52.284602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:52.284559    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:54.249550 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:54.249506    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:13:59.820677 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:13:59.820635    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:03.715587 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:03.715544    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"]
Sep 10 23:14:03.716088 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:03.715748    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f" podUID="7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" containerName="maas-api" containerID="cri-o://4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad" gracePeriod=30
Sep 10 23:14:03.743595 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:03.743557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:03.784652 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:03.784595    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"]
Sep 10 23:14:03.785167 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:03.785126    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc" podUID="83522d07-0473-46be-8257-333f664b33d0" containerName="payload-pre-processing" containerID="cri-o://7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece" gracePeriod=30
Sep 10 23:14:03.843159 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:03.843126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:03.941957 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:03.941864    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:04.034918 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.034893    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:14:04.042962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.042918    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:04.064062 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.064037    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:14:04.109599 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.109574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-metrics-certs\") pod \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") "
Sep 10 23:14:04.109752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.109624    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83522d07-0473-46be-8257-333f664b33d0-plugins-config-volume\") pod \"83522d07-0473-46be-8257-333f664b33d0\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") "
Sep 10 23:14:04.109752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.109684    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls\") pod \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") "
Sep 10 23:14:04.109752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.109706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6rct\" (UniqueName: \"kubernetes.io/projected/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-kube-api-access-w6rct\") pod \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") "
Sep 10 23:14:04.109752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.109740    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-openshift-service-ca\") pod \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\" (UID: \"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2\") "
Sep 10 23:14:04.109962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.109762    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g2g95\" (UniqueName: \"kubernetes.io/projected/83522d07-0473-46be-8257-333f664b33d0-kube-api-access-g2g95\") pod \"83522d07-0473-46be-8257-333f664b33d0\" (UID: \"83522d07-0473-46be-8257-333f664b33d0\") "
Sep 10 23:14:04.110137 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.110101    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83522d07-0473-46be-8257-333f664b33d0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "83522d07-0473-46be-8257-333f664b33d0" (UID: "83522d07-0473-46be-8257-333f664b33d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:14:04.110259 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.110169    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" (UID: "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:14:04.111834 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.111804    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" (UID: "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:14:04.111952 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.111875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-kube-api-access-w6rct" (OuterVolumeSpecName: "kube-api-access-w6rct") pod "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" (UID: "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2"). InnerVolumeSpecName "kube-api-access-w6rct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:04.111952 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.111883    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83522d07-0473-46be-8257-333f664b33d0-kube-api-access-g2g95" (OuterVolumeSpecName: "kube-api-access-g2g95") pod "83522d07-0473-46be-8257-333f664b33d0" (UID: "83522d07-0473-46be-8257-333f664b33d0"). InnerVolumeSpecName "kube-api-access-g2g95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:04.112123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.112104    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" (UID: "7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:14:04.205608 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.205573    2577 scope.go:117] "RemoveContainer" containerID="3b7cca724cc591d7589f2ace0eb073c105a988d2806d43db86425cc71c8230a7"
Sep 10 23:14:04.210811 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.210766    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/83522d07-0473-46be-8257-333f664b33d0-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:04.210811 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.210814    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:04.211014 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.210829    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6rct\" (UniqueName: \"kubernetes.io/projected/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-kube-api-access-w6rct\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:04.211014 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.210841    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:04.211014 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.210852    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2g95\" (UniqueName: \"kubernetes.io/projected/83522d07-0473-46be-8257-333f664b33d0-kube-api-access-g2g95\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:04.211014 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.210863    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:04.359867 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.359757    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" containerID="4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad" exitCode=0
Sep 10 23:14:04.359867 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.359817    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"
Sep 10 23:14:04.359867 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.359848    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f" event={"ID":"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2","Type":"ContainerDied","Data":"4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad"}
Sep 10 23:14:04.360143 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.359902    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f" event={"ID":"7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2","Type":"ContainerDied","Data":"d65b353d98701e25cd00ee0581d77cb62190f2f181f119e97609b77a414b5e23"}
Sep 10 23:14:04.360143 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.359928    2577 scope.go:117] "RemoveContainer" containerID="4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad"
Sep 10 23:14:04.368043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.368012    2577 generic.go:356] "Generic (PLEG): container finished" podID="83522d07-0473-46be-8257-333f664b33d0" containerID="7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece" exitCode=0
Sep 10 23:14:04.368164 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.368062    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"
Sep 10 23:14:04.368164 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.368094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc" event={"ID":"83522d07-0473-46be-8257-333f664b33d0","Type":"ContainerDied","Data":"7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece"}
Sep 10 23:14:04.368262 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.368190    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc" event={"ID":"83522d07-0473-46be-8257-333f664b33d0","Type":"ContainerDied","Data":"289c270ed417049aa235d7a038d85c5786700e3790a48ede8fed3c3eb63ff011"}
Sep 10 23:14:04.376993 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.376943    2577 scope.go:117] "RemoveContainer" containerID="4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad"
Sep 10 23:14:04.377211 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:04.377188    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad\": container with ID starting with 4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad not found: ID does not exist" containerID="4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad"
Sep 10 23:14:04.377270 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.377220    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad"} err="failed to get container status \"4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad\": rpc error: code = NotFound desc = could not find container \"4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad\": container with ID starting with 4f7db03bb86a7cfd0c5ff4dd9a27e68b4bfcedb010206abafb4416bfe22cc3ad not found: ID does not exist"
Sep 10 23:14:04.377270 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.377237    2577 scope.go:117] "RemoveContainer" containerID="7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece"
Sep 10 23:14:04.386413 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.386392    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"]
Sep 10 23:14:04.386544 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.386524    2577 scope.go:117] "RemoveContainer" containerID="7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece"
Sep 10 23:14:04.386758 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:04.386736    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece\": container with ID starting with 7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece not found: ID does not exist" containerID="7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece"
Sep 10 23:14:04.386827 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.386764    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece"} err="failed to get container status \"7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece\": rpc error: code = NotFound desc = could not find container \"7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece\": container with ID starting with 7fd8ea52e76a76421f36be899ae2e7afc7cee72e5792ef1495fecbb24cbc1ece not found: ID does not exist"
Sep 10 23:14:04.390919 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.390898    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-00575c78-598c4b5486-4wwqc"]
Sep 10 23:14:04.401910 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.401879    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"]
Sep 10 23:14:04.407878 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:04.407852    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-00575c78-75cf79d4c-zqk6f"]
Sep 10 23:14:05.106881 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:05.106775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:05.528503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:05.528468    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:05.841654 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:05.841570    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:06.121235 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:06.121134    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:06.215491 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:06.215458    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" path="/var/lib/kubelet/pods/7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2/volumes"
Sep 10 23:14:06.215900 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:06.215883    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83522d07-0473-46be-8257-333f664b33d0" path="/var/lib/kubelet/pods/83522d07-0473-46be-8257-333f664b33d0/volumes"
Sep 10 23:14:08.845672 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:08.845628    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:08.942301 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:08.942257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:09.983734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.983689    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"]
Sep 10 23:14:09.984196 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.984131    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83522d07-0473-46be-8257-333f664b33d0" containerName="payload-pre-processing"
Sep 10 23:14:09.984196 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.984147    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="83522d07-0473-46be-8257-333f664b33d0" containerName="payload-pre-processing"
Sep 10 23:14:09.984196 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.984162    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" containerName="maas-api"
Sep 10 23:14:09.984196 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.984167    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" containerName="maas-api"
Sep 10 23:14:09.984413 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.984243    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="83522d07-0473-46be-8257-333f664b33d0" containerName="payload-pre-processing"
Sep 10 23:14:09.984413 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.984254    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a1b4fe9-f127-4c4e-acfd-37e8f42a6dd2" containerName="maas-api"
Sep 10 23:14:09.989620 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.989597    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:09.991887 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.991866    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-e5ec8ad4-serving-cert\""
Sep 10 23:14:09.999086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:09.999065    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"]
Sep 10 23:14:10.025996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.025960    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.067565 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.067528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhtc7\" (UniqueName: \"kubernetes.io/projected/7bbd2816-161b-4165-9886-13ea961e897d-kube-api-access-lhtc7\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.067816 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.067606    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd2816-161b-4165-9886-13ea961e897d-openshift-service-ca\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.067816 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.067673    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.067816 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.067710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-metrics-certs\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.102287 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.102246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.168609 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.168560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhtc7\" (UniqueName: \"kubernetes.io/projected/7bbd2816-161b-4165-9886-13ea961e897d-kube-api-access-lhtc7\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.168923 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.168670    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd2816-161b-4165-9886-13ea961e897d-openshift-service-ca\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.168923 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.168709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.168923 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.168843    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-metrics-certs\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.168923 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:10.168905    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-serving-cert: secret "maas-api-e2e-derive-e5ec8ad4-serving-cert" not found
Sep 10 23:14:10.169140 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:10.168975    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls podName:7bbd2816-161b-4165-9886-13ea961e897d nodeName:}" failed. No retries permitted until 2026-09-10 23:14:10.668953292 +0000 UTC m=+967.105008053 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls") pod "maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc" (UID: "7bbd2816-161b-4165-9886-13ea961e897d") : secret "maas-api-e2e-derive-e5ec8ad4-serving-cert" not found
Sep 10 23:14:10.169444 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.169426    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd2816-161b-4165-9886-13ea961e897d-openshift-service-ca\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.171241 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.171224    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-metrics-certs\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.180035 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.180011    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhtc7\" (UniqueName: \"kubernetes.io/projected/7bbd2816-161b-4165-9886-13ea961e897d-kube-api-access-lhtc7\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.322353 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.322258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.575485 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.575392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.664421 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.664387    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:10.673109 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.673076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.675557 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.675525    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls\") pod \"maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.853379 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.853301    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"]
Sep 10 23:14:10.857032 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.857016    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:10.862905 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.862629    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-e5ec8ad4\""
Sep 10 23:14:10.864336 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.864142    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-e5ec8ad4-dockercfg-5jt6n\""
Sep 10 23:14:10.883506 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.883475    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"]
Sep 10 23:14:10.902125 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.902097    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:10.976102 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.976069    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-plugins-config-volume\") pod \"payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") " pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:10.976251 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:10.976124    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4db9\" (UniqueName: \"kubernetes.io/projected/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-kube-api-access-n4db9\") pod \"payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") " pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:11.077501 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.077464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-plugins-config-volume\") pod \"payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") " pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:11.077906 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.077523    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4db9\" (UniqueName: \"kubernetes.io/projected/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-kube-api-access-n4db9\") pod \"payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") " pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:11.078283 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.078260    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-plugins-config-volume\") pod \"payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") " pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:11.086930 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.086903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4db9\" (UniqueName: \"kubernetes.io/projected/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-kube-api-access-n4db9\") pod \"payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") " pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:11.169739 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.169698    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:11.178117 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.177958    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"]
Sep 10 23:14:11.181253 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:14:11.181228    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbd2816_161b_4165_9886_13ea961e897d.slice/crio-8a853e79cddbcac2a5e054a86ab45109d1ac29991250ddf2b1dd1011abb3da03 WatchSource:0}: Error finding container 8a853e79cddbcac2a5e054a86ab45109d1ac29991250ddf2b1dd1011abb3da03: Status 404 returned error can't find the container with id 8a853e79cddbcac2a5e054a86ab45109d1ac29991250ddf2b1dd1011abb3da03
Sep 10 23:14:11.405160 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.405121    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc" event={"ID":"7bbd2816-161b-4165-9886-13ea961e897d","Type":"ContainerStarted","Data":"8a853e79cddbcac2a5e054a86ab45109d1ac29991250ddf2b1dd1011abb3da03"}
Sep 10 23:14:11.452289 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:11.452261    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"]
Sep 10 23:14:11.454890 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:14:11.454864    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbde65c_7b18_4094_9a6f_5cbb5797ff52.slice/crio-d2f1c9acf33bd0ecca5581a2c12e1dc93b3ad8f8ba0678dd14bc27ad35b2ccda WatchSource:0}: Error finding container d2f1c9acf33bd0ecca5581a2c12e1dc93b3ad8f8ba0678dd14bc27ad35b2ccda: Status 404 returned error can't find the container with id d2f1c9acf33bd0ecca5581a2c12e1dc93b3ad8f8ba0678dd14bc27ad35b2ccda
Sep 10 23:14:12.411633 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:12.411594    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc" event={"ID":"7bbd2816-161b-4165-9886-13ea961e897d","Type":"ContainerStarted","Data":"347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98"}
Sep 10 23:14:12.412390 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:12.411765    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:12.413244 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:12.413220    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p" event={"ID":"7fbde65c-7b18-4094-9a6f-5cbb5797ff52","Type":"ContainerStarted","Data":"b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b"}
Sep 10 23:14:12.413374 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:12.413266    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p" event={"ID":"7fbde65c-7b18-4094-9a6f-5cbb5797ff52","Type":"ContainerStarted","Data":"d2f1c9acf33bd0ecca5581a2c12e1dc93b3ad8f8ba0678dd14bc27ad35b2ccda"}
Sep 10 23:14:12.413374 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:12.413299    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:12.437140 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:12.437082    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc" podStartSLOduration=3.016760366 podStartE2EDuration="3.43706723s" totalImagesPullingTime="420.306864ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:12.436309745 +0000 UTC m=+968.872364518" watchObservedRunningTime="2026-09-10 23:14:12.43706723 +0000 UTC m=+968.873122003"
Sep 10 23:14:12.458169 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:12.458105    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p" podStartSLOduration=2.458080723 podStartE2EDuration="2.458080723s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:12.456391662 +0000 UTC m=+968.892446434" watchObservedRunningTime="2026-09-10 23:14:12.458080723 +0000 UTC m=+968.894135501"
Sep 10 23:14:18.423753 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:18.423722    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:23.420946 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:23.420913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:34.894594 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:34.894507    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:37.061567 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:37.061530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:40.318845 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:40.318806    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:40.916900 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:40.916857    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"]
Sep 10 23:14:40.917148 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:40.917106    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc" podUID="7bbd2816-161b-4165-9886-13ea961e897d" containerName="maas-api" containerID="cri-o://347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98" gracePeriod=30
Sep 10 23:14:40.930679 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:40.930646    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:40.967230 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:40.967197    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"]
Sep 10 23:14:40.967457 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:40.967428    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p" podUID="7fbde65c-7b18-4094-9a6f-5cbb5797ff52" containerName="payload-processing" containerID="cri-o://b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b" gracePeriod=30
Sep 10 23:14:41.031508 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.031474    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.130818 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.130670    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.197088 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.197057    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.212127 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.212088    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:41.249533 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.249506    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:41.258969 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.258942    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls\") pod \"7bbd2816-161b-4165-9886-13ea961e897d\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") "
Sep 10 23:14:41.259123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.258995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd2816-161b-4165-9886-13ea961e897d-openshift-service-ca\") pod \"7bbd2816-161b-4165-9886-13ea961e897d\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") "
Sep 10 23:14:41.259123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.259103    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-metrics-certs\") pod \"7bbd2816-161b-4165-9886-13ea961e897d\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") "
Sep 10 23:14:41.259255 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.259175    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhtc7\" (UniqueName: \"kubernetes.io/projected/7bbd2816-161b-4165-9886-13ea961e897d-kube-api-access-lhtc7\") pod \"7bbd2816-161b-4165-9886-13ea961e897d\" (UID: \"7bbd2816-161b-4165-9886-13ea961e897d\") "
Sep 10 23:14:41.259376 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.259353    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbd2816-161b-4165-9886-13ea961e897d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7bbd2816-161b-4165-9886-13ea961e897d" (UID: "7bbd2816-161b-4165-9886-13ea961e897d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:14:41.260032 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.259580    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbd2816-161b-4165-9886-13ea961e897d-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:41.261428 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.261407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7bbd2816-161b-4165-9886-13ea961e897d" (UID: "7bbd2816-161b-4165-9886-13ea961e897d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:14:41.261750 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.261730    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bbd2816-161b-4165-9886-13ea961e897d-kube-api-access-lhtc7" (OuterVolumeSpecName: "kube-api-access-lhtc7") pod "7bbd2816-161b-4165-9886-13ea961e897d" (UID: "7bbd2816-161b-4165-9886-13ea961e897d"). InnerVolumeSpecName "kube-api-access-lhtc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:41.261841 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.261738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7bbd2816-161b-4165-9886-13ea961e897d" (UID: "7bbd2816-161b-4165-9886-13ea961e897d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:14:41.308025 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.307983    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:41.360142 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.360105    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-plugins-config-volume\") pod \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") "
Sep 10 23:14:41.360522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.360171    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4db9\" (UniqueName: \"kubernetes.io/projected/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-kube-api-access-n4db9\") pod \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\" (UID: \"7fbde65c-7b18-4094-9a6f-5cbb5797ff52\") "
Sep 10 23:14:41.360522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.360388    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:41.360522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.360401    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhtc7\" (UniqueName: \"kubernetes.io/projected/7bbd2816-161b-4165-9886-13ea961e897d-kube-api-access-lhtc7\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:41.360522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.360411    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7bbd2816-161b-4165-9886-13ea961e897d-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:41.360522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.360502    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7fbde65c-7b18-4094-9a6f-5cbb5797ff52" (UID: "7fbde65c-7b18-4094-9a6f-5cbb5797ff52"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:14:41.362261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.362230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-kube-api-access-n4db9" (OuterVolumeSpecName: "kube-api-access-n4db9") pod "7fbde65c-7b18-4094-9a6f-5cbb5797ff52" (UID: "7fbde65c-7b18-4094-9a6f-5cbb5797ff52"). InnerVolumeSpecName "kube-api-access-n4db9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:41.461981 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.461886    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:41.461981 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.461919    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4db9\" (UniqueName: \"kubernetes.io/projected/7fbde65c-7b18-4094-9a6f-5cbb5797ff52-kube-api-access-n4db9\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:41.554528 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.554494    2577 generic.go:356] "Generic (PLEG): container finished" podID="7bbd2816-161b-4165-9886-13ea961e897d" containerID="347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98" exitCode=0
Sep 10 23:14:41.554710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.554540    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"
Sep 10 23:14:41.554710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.554582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc" event={"ID":"7bbd2816-161b-4165-9886-13ea961e897d","Type":"ContainerDied","Data":"347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98"}
Sep 10 23:14:41.554710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.554636    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc" event={"ID":"7bbd2816-161b-4165-9886-13ea961e897d","Type":"ContainerDied","Data":"8a853e79cddbcac2a5e054a86ab45109d1ac29991250ddf2b1dd1011abb3da03"}
Sep 10 23:14:41.554710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.554667    2577 scope.go:117] "RemoveContainer" containerID="347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98"
Sep 10 23:14:41.555995 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.555933    2577 generic.go:356] "Generic (PLEG): container finished" podID="7fbde65c-7b18-4094-9a6f-5cbb5797ff52" containerID="b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b" exitCode=0
Sep 10 23:14:41.555995 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.555964    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p" event={"ID":"7fbde65c-7b18-4094-9a6f-5cbb5797ff52","Type":"ContainerDied","Data":"b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b"}
Sep 10 23:14:41.556132 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.556025    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p" event={"ID":"7fbde65c-7b18-4094-9a6f-5cbb5797ff52","Type":"ContainerDied","Data":"d2f1c9acf33bd0ecca5581a2c12e1dc93b3ad8f8ba0678dd14bc27ad35b2ccda"}
Sep 10 23:14:41.556132 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.555981    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"
Sep 10 23:14:41.566189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.566166    2577 scope.go:117] "RemoveContainer" containerID="347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98"
Sep 10 23:14:41.566523 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:41.566504    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98\": container with ID starting with 347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98 not found: ID does not exist" containerID="347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98"
Sep 10 23:14:41.566605 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.566531    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98"} err="failed to get container status \"347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98\": rpc error: code = NotFound desc = could not find container \"347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98\": container with ID starting with 347dc791d58969e5eec99e6458de170c0b33ec56f0b9e3047300b09383a11a98 not found: ID does not exist"
Sep 10 23:14:41.566605 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.566548    2577 scope.go:117] "RemoveContainer" containerID="b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b"
Sep 10 23:14:41.574703 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.574684    2577 scope.go:117] "RemoveContainer" containerID="b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b"
Sep 10 23:14:41.574987 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:41.574967    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b\": container with ID starting with b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b not found: ID does not exist" containerID="b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b"
Sep 10 23:14:41.575061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.574997    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b"} err="failed to get container status \"b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b\": rpc error: code = NotFound desc = could not find container \"b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b\": container with ID starting with b813a878fc051bdcf155e732be5bc2d7648e2112baecdeadccf2de8373cf126b not found: ID does not exist"
Sep 10 23:14:41.582028 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.582006    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"]
Sep 10 23:14:41.588373 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.588349    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-e5ec8ad4-79f5f77657-q6z9p"]
Sep 10 23:14:41.601090 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.601063    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"]
Sep 10 23:14:41.602680 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.602660    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-e5ec8ad4-dcddf75cd-ks5bc"]
Sep 10 23:14:41.978386 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:41.978337    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:42.079225 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:42.079192    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:42.179036 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:42.178999    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:42.215163 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:42.215087    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7bbd2816-161b-4165-9886-13ea961e897d" path="/var/lib/kubelet/pods/7bbd2816-161b-4165-9886-13ea961e897d/volumes"
Sep 10 23:14:42.215452 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:42.215439    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbde65c-7b18-4094-9a6f-5cbb5797ff52" path="/var/lib/kubelet/pods/7fbde65c-7b18-4094-9a6f-5cbb5797ff52/volumes"
Sep 10 23:14:43.498752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:43.498707    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:44.527357 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:44.527324    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.179447 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.179409    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"]
Sep 10 23:14:45.179639 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.179615    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6" podUID="6a9156d2-3e81-451c-8ac5-317a50da2e3c" containerName="maas-api" containerID="cri-o://b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8" gracePeriod=30
Sep 10 23:14:45.215557 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.215498    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.253041 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.253009    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"]
Sep 10 23:14:45.253257 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.253232    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw" podUID="b5524b8f-1944-46fc-9e33-fa3733f2d647" containerName="payload-processing" containerID="cri-o://7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a" gracePeriod=30
Sep 10 23:14:45.296367 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.296332    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.357318 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.357280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.458395 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.458362    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.501203 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.501180    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:14:45.519324 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.519203    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.538020 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.537994    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:14:45.577972 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.577933    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5524b8f-1944-46fc-9e33-fa3733f2d647" containerID="7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a" exitCode=0
Sep 10 23:14:45.578151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.577979    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"
Sep 10 23:14:45.578151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.578022    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw" event={"ID":"b5524b8f-1944-46fc-9e33-fa3733f2d647","Type":"ContainerDied","Data":"7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a"}
Sep 10 23:14:45.578151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.578090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw" event={"ID":"b5524b8f-1944-46fc-9e33-fa3733f2d647","Type":"ContainerDied","Data":"bd883ea6e9a0acde13af6ebad83d8c0017627e6084a9dc16b91da3578b09c2a0"}
Sep 10 23:14:45.578151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.578122    2577 scope.go:117] "RemoveContainer" containerID="7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a"
Sep 10 23:14:45.579514 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.579494    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a9156d2-3e81-451c-8ac5-317a50da2e3c" containerID="b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8" exitCode=0
Sep 10 23:14:45.579627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.579527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6" event={"ID":"6a9156d2-3e81-451c-8ac5-317a50da2e3c","Type":"ContainerDied","Data":"b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8"}
Sep 10 23:14:45.579627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.579553    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"
Sep 10 23:14:45.579627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.579575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6" event={"ID":"6a9156d2-3e81-451c-8ac5-317a50da2e3c","Type":"ContainerDied","Data":"5f28db15a90b4118ae53e8679320bb4c8ab39b0cdbc36b399436a1a7578eba11"}
Sep 10 23:14:45.587098 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.587080    2577 scope.go:117] "RemoveContainer" containerID="7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a"
Sep 10 23:14:45.587424 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:45.587404    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a\": container with ID starting with 7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a not found: ID does not exist" containerID="7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a"
Sep 10 23:14:45.587498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.587436    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a"} err="failed to get container status \"7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a\": rpc error: code = NotFound desc = could not find container \"7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a\": container with ID starting with 7d044b48fd82e8a0e2073183db03f4c7559af7ba92b7692cf578b36a4aa0ed5a not found: ID does not exist"
Sep 10 23:14:45.587498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.587459    2577 scope.go:117] "RemoveContainer" containerID="b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8"
Sep 10 23:14:45.595564 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.595536    2577 scope.go:117] "RemoveContainer" containerID="b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8"
Sep 10 23:14:45.595846 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:14:45.595829    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8\": container with ID starting with b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8 not found: ID does not exist" containerID="b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8"
Sep 10 23:14:45.595918 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.595854    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8"} err="failed to get container status \"b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8\": rpc error: code = NotFound desc = could not find container \"b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8\": container with ID starting with b141c110040f820dc2c51430ee99f241fb8fdae2889d81c67d15d0550c6e05c8 not found: ID does not exist"
Sep 10 23:14:45.597090 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls\") pod \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") "
Sep 10 23:14:45.597140 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597119    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c9bld\" (UniqueName: \"kubernetes.io/projected/b5524b8f-1944-46fc-9e33-fa3733f2d647-kube-api-access-c9bld\") pod \"b5524b8f-1944-46fc-9e33-fa3733f2d647\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") "
Sep 10 23:14:45.597179 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597157    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w47jf\" (UniqueName: \"kubernetes.io/projected/6a9156d2-3e81-451c-8ac5-317a50da2e3c-kube-api-access-w47jf\") pod \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") "
Sep 10 23:14:45.597216 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597188    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5524b8f-1944-46fc-9e33-fa3733f2d647-plugins-config-volume\") pod \"b5524b8f-1944-46fc-9e33-fa3733f2d647\" (UID: \"b5524b8f-1944-46fc-9e33-fa3733f2d647\") "
Sep 10 23:14:45.597252 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9156d2-3e81-451c-8ac5-317a50da2e3c-openshift-service-ca\") pod \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") "
Sep 10 23:14:45.597298 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597287    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-metrics-certs\") pod \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\" (UID: \"6a9156d2-3e81-451c-8ac5-317a50da2e3c\") "
Sep 10 23:14:45.597653 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597628    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9156d2-3e81-451c-8ac5-317a50da2e3c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6a9156d2-3e81-451c-8ac5-317a50da2e3c" (UID: "6a9156d2-3e81-451c-8ac5-317a50da2e3c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:14:45.597653 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.597634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5524b8f-1944-46fc-9e33-fa3733f2d647-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5524b8f-1944-46fc-9e33-fa3733f2d647" (UID: "b5524b8f-1944-46fc-9e33-fa3733f2d647"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:14:45.599230 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.599204    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5524b8f-1944-46fc-9e33-fa3733f2d647-kube-api-access-c9bld" (OuterVolumeSpecName: "kube-api-access-c9bld") pod "b5524b8f-1944-46fc-9e33-fa3733f2d647" (UID: "b5524b8f-1944-46fc-9e33-fa3733f2d647"). InnerVolumeSpecName "kube-api-access-c9bld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:45.599335 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.599241    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6a9156d2-3e81-451c-8ac5-317a50da2e3c" (UID: "6a9156d2-3e81-451c-8ac5-317a50da2e3c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:14:45.599609 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.599591    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6a9156d2-3e81-451c-8ac5-317a50da2e3c" (UID: "6a9156d2-3e81-451c-8ac5-317a50da2e3c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:14:45.599669 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.599596    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a9156d2-3e81-451c-8ac5-317a50da2e3c-kube-api-access-w47jf" (OuterVolumeSpecName: "kube-api-access-w47jf") pod "6a9156d2-3e81-451c-8ac5-317a50da2e3c" (UID: "6a9156d2-3e81-451c-8ac5-317a50da2e3c"). InnerVolumeSpecName "kube-api-access-w47jf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:14:45.698895 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.698862    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:45.698895 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.698888    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6a9156d2-3e81-451c-8ac5-317a50da2e3c-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:45.698895 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.698897    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9bld\" (UniqueName: \"kubernetes.io/projected/b5524b8f-1944-46fc-9e33-fa3733f2d647-kube-api-access-c9bld\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:45.699122 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.698910    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w47jf\" (UniqueName: \"kubernetes.io/projected/6a9156d2-3e81-451c-8ac5-317a50da2e3c-kube-api-access-w47jf\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:45.699122 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.698920    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5524b8f-1944-46fc-9e33-fa3733f2d647-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:45.699122 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.698928    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9156d2-3e81-451c-8ac5-317a50da2e3c-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:14:45.807882 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.807762    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:45.904250 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.904216    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"]
Sep 10 23:14:45.910507 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.910482    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-autores-3a438f-596f9d4cd-z86q6"]
Sep 10 23:14:45.922079 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.922049    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"]
Sep 10 23:14:45.926138 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:45.926113    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-3a438f-74ff884cb7-8pnrw"]
Sep 10 23:14:46.214722 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:46.214689    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9156d2-3e81-451c-8ac5-317a50da2e3c" path="/var/lib/kubelet/pods/6a9156d2-3e81-451c-8ac5-317a50da2e3c/volumes"
Sep 10 23:14:46.215101 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:46.215087    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5524b8f-1944-46fc-9e33-fa3733f2d647" path="/var/lib/kubelet/pods/b5524b8f-1944-46fc-9e33-fa3733f2d647/volumes"
Sep 10 23:14:49.832166 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:49.832126    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:49.902987 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:49.902948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:49.993932 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:49.993887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:50.092850 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:50.092744    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:50.505650 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:50.505617    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:50.890285 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:50.890163    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:57.167265 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167231    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"]
Sep 10 23:14:57.167609 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167596    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9156d2-3e81-451c-8ac5-317a50da2e3c" containerName="maas-api"
Sep 10 23:14:57.167609 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167607    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9156d2-3e81-451c-8ac5-317a50da2e3c" containerName="maas-api"
Sep 10 23:14:57.167689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167620    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bbd2816-161b-4165-9886-13ea961e897d" containerName="maas-api"
Sep 10 23:14:57.167689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167625    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bbd2816-161b-4165-9886-13ea961e897d" containerName="maas-api"
Sep 10 23:14:57.167689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167635    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbde65c-7b18-4094-9a6f-5cbb5797ff52" containerName="payload-processing"
Sep 10 23:14:57.167689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167640    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbde65c-7b18-4094-9a6f-5cbb5797ff52" containerName="payload-processing"
Sep 10 23:14:57.167689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167647    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5524b8f-1944-46fc-9e33-fa3733f2d647" containerName="payload-processing"
Sep 10 23:14:57.167689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167652    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5524b8f-1944-46fc-9e33-fa3733f2d647" containerName="payload-processing"
Sep 10 23:14:57.167877 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167710    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5524b8f-1944-46fc-9e33-fa3733f2d647" containerName="payload-processing"
Sep 10 23:14:57.167877 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167718    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bbd2816-161b-4165-9886-13ea961e897d" containerName="maas-api"
Sep 10 23:14:57.167877 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167724    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbde65c-7b18-4094-9a6f-5cbb5797ff52" containerName="payload-processing"
Sep 10 23:14:57.167877 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.167730    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9156d2-3e81-451c-8ac5-317a50da2e3c" containerName="maas-api"
Sep 10 23:14:57.172170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.172151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.174554 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.174533    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28dbae\"/\"openshift-service-ca.crt\""
Sep 10 23:14:57.175280 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.175254    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-28dbae\"/\"kube-root-ca.crt\""
Sep 10 23:14:57.175406 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.175306    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28dbae\"/\"default-dockercfg-rpn6r\""
Sep 10 23:14:57.175406 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.175326    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-28dbae\"/\"test-model-28dbae-kserve-self-signed-certs\""
Sep 10 23:14:57.179156 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.179021    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"]
Sep 10 23:14:57.191554 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.191521    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:14:57.307986 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.307946    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-tmp-dir\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.307986 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.307992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc8f99-4270-4dad-94b6-2fc713703918-tls-certs\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.308308 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.308042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-model-cache\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.308308 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.308061    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-dshm\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.308308 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.308191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-home\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.308308 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.308263    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprq7\" (UniqueName: \"kubernetes.io/projected/afcc8f99-4270-4dad-94b6-2fc713703918-kube-api-access-pprq7\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.409737 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.409684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-dshm\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.409931 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.409768    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-home\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.409931 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.409857    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprq7\" (UniqueName: \"kubernetes.io/projected/afcc8f99-4270-4dad-94b6-2fc713703918-kube-api-access-pprq7\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.409931 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.409896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-tmp-dir\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.410066 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.409928    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc8f99-4270-4dad-94b6-2fc713703918-tls-certs\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.410066 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.409974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-model-cache\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.410239 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.410216    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-home\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.410309 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.410272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-tmp-dir\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.410309 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.410302    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-model-cache\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.412043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.412021    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-dshm\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.412372 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.412354    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc8f99-4270-4dad-94b6-2fc713703918-tls-certs\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.418710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.418645    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprq7\" (UniqueName: \"kubernetes.io/projected/afcc8f99-4270-4dad-94b6-2fc713703918-kube-api-access-pprq7\") pod \"test-model-28dbae-kserve-5fc9768ffc-qh4gg\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") " pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.481538 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.481496    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:14:57.756145 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:57.756121    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"]
Sep 10 23:14:57.761691 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:14:57.761595    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podafcc8f99_4270_4dad_94b6_2fc713703918.slice/crio-235ecb67402e161d58956ec201ad758f665be137f5e65b4821ed25bc61b3fdb4 WatchSource:0}: Error finding container 235ecb67402e161d58956ec201ad758f665be137f5e65b4821ed25bc61b3fdb4: Status 404 returned error can't find the container with id 235ecb67402e161d58956ec201ad758f665be137f5e65b4821ed25bc61b3fdb4
Sep 10 23:14:58.647475 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:58.647440    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg" event={"ID":"afcc8f99-4270-4dad-94b6-2fc713703918","Type":"ContainerStarted","Data":"e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0"}
Sep 10 23:14:58.647475 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:58.647482    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg" event={"ID":"afcc8f99-4270-4dad-94b6-2fc713703918","Type":"ContainerStarted","Data":"235ecb67402e161d58956ec201ad758f665be137f5e65b4821ed25bc61b3fdb4"}
Sep 10 23:14:58.668175 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:14:58.668122    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg" podStartSLOduration=1.668106153 podStartE2EDuration="1.668106153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:14:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:14:58.666111381 +0000 UTC m=+1015.102166234" watchObservedRunningTime="2026-09-10 23:14:58.668106153 +0000 UTC m=+1015.104160924"
Sep 10 23:15:07.481839 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:07.481773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:15:07.482362 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:07.481868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:15:07.494466 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:07.494433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:15:07.701209 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:07.701179    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:15:15.241854 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:15.241817    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:16.431124 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:16.431084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:20.383306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:20.383262    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:22.824998 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:22.824960    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:22.926424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:22.926392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:23.026676 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:23.026630    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:23.788244 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:23.788189    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:27.194036 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:27.193987    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:27.908033 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:27.907995    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:28.159369 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:28.159278    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:28.581551 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:28.581454    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:28.878654 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:28.878555    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:28.941971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:28.941942    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:31.950906 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:31.950866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:32.047953 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:32.047915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:33.081114 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.081079    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"]
Sep 10 23:15:33.084650 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.084628    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.086914 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.086891    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-8972e2-serving-cert\""
Sep 10 23:15:33.097096 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.097030    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"]
Sep 10 23:15:33.129760 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.129721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-openshift-service-ca\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.129945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.129791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-metrics-certs\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.129945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.129858    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.130023 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.129938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8nplw\" (UniqueName: \"kubernetes.io/projected/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-kube-api-access-8nplw\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.162702 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.162659    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:33.230496 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.230453    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-metrics-certs\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.230711 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.230510    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.230711 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.230573    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8nplw\" (UniqueName: \"kubernetes.io/projected/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-kube-api-access-8nplw\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.230711 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.230652    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-openshift-service-ca\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.230939 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:15:33.230826    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-serving-cert: secret "maas-api-e2e-explicit-8972e2-serving-cert" not found
Sep 10 23:15:33.230939 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:15:33.230916    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls podName:36dae147-4ecc-43a3-840f-f2fd0fe9b56a nodeName:}" failed. No retries permitted until 2026-09-10 23:15:33.730893534 +0000 UTC m=+1050.166948294 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls") pod "maas-api-e2e-explicit-8972e2-c594546f-d6m96" (UID: "36dae147-4ecc-43a3-840f-f2fd0fe9b56a") : secret "maas-api-e2e-explicit-8972e2-serving-cert" not found
Sep 10 23:15:33.231656 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.231634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-openshift-service-ca\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.233296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.233278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-metrics-certs\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.245153 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.245121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8nplw\" (UniqueName: \"kubernetes.io/projected/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-kube-api-access-8nplw\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.272250 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.272211    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:33.459788 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.459735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:33.685966 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.685929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:33.734454 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.734367    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.736985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.736957    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls\") pod \"maas-api-e2e-explicit-8972e2-c594546f-d6m96\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:33.783533 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.783494    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:33.996271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:33.996177    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:34.274075 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:34.274011    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"]
Sep 10 23:15:34.276437 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:15:34.276409    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36dae147_4ecc_43a3_840f_f2fd0fe9b56a.slice/crio-8d78dc017d8cb0e801d7c510935c709da492d1812ba48262a285f2c4fbe53c3f WatchSource:0}: Error finding container 8d78dc017d8cb0e801d7c510935c709da492d1812ba48262a285f2c4fbe53c3f: Status 404 returned error can't find the container with id 8d78dc017d8cb0e801d7c510935c709da492d1812ba48262a285f2c4fbe53c3f
Sep 10 23:15:34.277852 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:34.277833    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:15:34.818569 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:34.818527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96" event={"ID":"36dae147-4ecc-43a3-840f-f2fd0fe9b56a","Type":"ContainerStarted","Data":"bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0"}
Sep 10 23:15:34.818767 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:34.818582    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96" event={"ID":"36dae147-4ecc-43a3-840f-f2fd0fe9b56a","Type":"ContainerStarted","Data":"8d78dc017d8cb0e801d7c510935c709da492d1812ba48262a285f2c4fbe53c3f"}
Sep 10 23:15:34.818767 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:34.818684    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:34.838403 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:34.838314    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96" podStartSLOduration=1.49157063 podStartE2EDuration="1.838292694s" totalImagesPullingTime="346.722064ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:15:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:15:34.837816701 +0000 UTC m=+1051.273871476" watchObservedRunningTime="2026-09-10 23:15:34.838292694 +0000 UTC m=+1051.274347467"
Sep 10 23:15:37.558522 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:37.558482    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:38.859624 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:38.859588    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:40.828409 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:40.828375    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:15:45.413515 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:45.413443    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:45.476216 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:45.476178    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:45.555254 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:45.555217    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:45.648627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:45.648591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:56.123091 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:56.123010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:56.891817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:56.891754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:15:56.993190 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:15:56.993145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:03.170240 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:03.170193    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:03.270570 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:03.270525    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:03.370541 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:03.370505    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:03.472211 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:03.472118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:04.104537 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:04.104499    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:04.203437 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:04.203399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:04.304669 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:04.304615    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:05.686569 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:05.686534    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:05.742822 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:05.742764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:07.811945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:07.811896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:09.963695 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:09.963647    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:12.515015 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:12.514977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:12.614920 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:12.614882    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:13.175328 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:13.175289    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:18.441001 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:18.440950    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:19.835287 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:19.835247    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:19.937357 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:19.937321    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:20.059682 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:20.059636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:20.354283 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:20.354245    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:21.146887 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:21.146846    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:23.339146 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:23.339107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:23.425450 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:23.425414    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:24.478379 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.478342    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"]
Sep 10 23:16:24.482416 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.482387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.485494 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.485465    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-1f90a11e-serving-cert\""
Sep 10 23:16:24.494181 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.494140    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-maas-api-tls\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.494406 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.494386    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-metrics-certs\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.494557 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.494533    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lgkwx\" (UniqueName: \"kubernetes.io/projected/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-kube-api-access-lgkwx\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.494671 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.494656    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-openshift-service-ca\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.498213 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.498186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"]
Sep 10 23:16:24.524474 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.524433    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:24.595084 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.595044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-maas-api-tls\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.595084 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.595086    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-metrics-certs\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.595370 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.595110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lgkwx\" (UniqueName: \"kubernetes.io/projected/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-kube-api-access-lgkwx\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.595458 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.595427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-openshift-service-ca\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.596040 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.596012    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-openshift-service-ca\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.597830 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.597804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-metrics-certs\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.597944 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.597804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-maas-api-tls\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.606027 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.605995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lgkwx\" (UniqueName: \"kubernetes.io/projected/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-kube-api-access-lgkwx\") pod \"maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:24.623829 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.623794    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:24.724061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.724022    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:24.797346 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:24.797244    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:25.075686 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.075661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"]
Sep 10 23:16:25.077910 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:16:25.077871    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bad6c63_0989_4fe5_9d63_1f415df8a5a3.slice/crio-5b26b46221c0d91c10fde2e5955e0aacd8b178883576efc575f2cbf96321f102 WatchSource:0}: Error finding container 5b26b46221c0d91c10fde2e5955e0aacd8b178883576efc575f2cbf96321f102: Status 404 returned error can't find the container with id 5b26b46221c0d91c10fde2e5955e0aacd8b178883576efc575f2cbf96321f102
Sep 10 23:16:25.080227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.080196    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:25.180362 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.180328    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:25.318957 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.318828    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"]
Sep 10 23:16:25.323620 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.323593    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.326170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.326105    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-1f90a11e\""
Sep 10 23:16:25.326306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.326182    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-1f90a11e-dockercfg-4qd29\""
Sep 10 23:16:25.334194 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.334169    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"]
Sep 10 23:16:25.403267 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.403231    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c347cade-5767-4f3b-96a7-a835e62f5cd5-plugins-config-volume\") pod \"payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") " pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.403453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.403277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpmd4\" (UniqueName: \"kubernetes.io/projected/c347cade-5767-4f3b-96a7-a835e62f5cd5-kube-api-access-wpmd4\") pod \"payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") " pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.503889 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.503864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c347cade-5767-4f3b-96a7-a835e62f5cd5-plugins-config-volume\") pod \"payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") " pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.504318 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.503902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpmd4\" (UniqueName: \"kubernetes.io/projected/c347cade-5767-4f3b-96a7-a835e62f5cd5-kube-api-access-wpmd4\") pod \"payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") " pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.504624 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.504602    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c347cade-5767-4f3b-96a7-a835e62f5cd5-plugins-config-volume\") pod \"payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") " pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.513657 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.513634    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpmd4\" (UniqueName: \"kubernetes.io/projected/c347cade-5767-4f3b-96a7-a835e62f5cd5-kube-api-access-wpmd4\") pod \"payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") " pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.635602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.635503    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:25.928818 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:25.928768    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"]
Sep 10 23:16:25.932828 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:16:25.932802    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc347cade_5767_4f3b_96a7_a835e62f5cd5.slice/crio-925659423e87214594b8642c799bad70cbc2de007378d70610b3a51c372899f9 WatchSource:0}: Error finding container 925659423e87214594b8642c799bad70cbc2de007378d70610b3a51c372899f9: Status 404 returned error can't find the container with id 925659423e87214594b8642c799bad70cbc2de007378d70610b3a51c372899f9
Sep 10 23:16:26.065497 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.065456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb" event={"ID":"6bad6c63-0989-4fe5-9d63-1f415df8a5a3","Type":"ContainerStarted","Data":"563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09"}
Sep 10 23:16:26.065691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.065525    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb" event={"ID":"6bad6c63-0989-4fe5-9d63-1f415df8a5a3","Type":"ContainerStarted","Data":"5b26b46221c0d91c10fde2e5955e0aacd8b178883576efc575f2cbf96321f102"}
Sep 10 23:16:26.065691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.065577    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:26.067506 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.067481    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd" event={"ID":"c347cade-5767-4f3b-96a7-a835e62f5cd5","Type":"ContainerStarted","Data":"01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043"}
Sep 10 23:16:26.067600 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.067527    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:26.067600 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.067551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd" event={"ID":"c347cade-5767-4f3b-96a7-a835e62f5cd5","Type":"ContainerStarted","Data":"925659423e87214594b8642c799bad70cbc2de007378d70610b3a51c372899f9"}
Sep 10 23:16:26.095505 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.095451    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb" podStartSLOduration=1.723240662 podStartE2EDuration="2.095433423s" totalImagesPullingTime="372.192761ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:26.09288324 +0000 UTC m=+1102.528938015" watchObservedRunningTime="2026-09-10 23:16:26.095433423 +0000 UTC m=+1102.531488269"
Sep 10 23:16:26.120042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:26.119993    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd" podStartSLOduration=1.119978199 podStartE2EDuration="1.119978199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:16:26.117896361 +0000 UTC m=+1102.553951136" watchObservedRunningTime="2026-09-10 23:16:26.119978199 +0000 UTC m=+1102.556032972"
Sep 10 23:16:28.697164 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:28.697121    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:29.275428 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.275394    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"]
Sep 10 23:16:29.275642 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.275613    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96" podUID="36dae147-4ecc-43a3-840f-f2fd0fe9b56a" containerName="maas-api" containerID="cri-o://bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0" gracePeriod=30
Sep 10 23:16:29.299668 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.299636    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:29.376556 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.376523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:29.432465 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.432426    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:29.490618 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.490584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:29.537348 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.537278    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:16:29.544347 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.544312    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:29.643153 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.643115    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-openshift-service-ca\") pod \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") "
Sep 10 23:16:29.643313 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.643171    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-metrics-certs\") pod \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") "
Sep 10 23:16:29.643313 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.643297    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8nplw\" (UniqueName: \"kubernetes.io/projected/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-kube-api-access-8nplw\") pod \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") "
Sep 10 23:16:29.643406 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.643372    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls\") pod \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\" (UID: \"36dae147-4ecc-43a3-840f-f2fd0fe9b56a\") "
Sep 10 23:16:29.643532 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.643513    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "36dae147-4ecc-43a3-840f-f2fd0fe9b56a" (UID: "36dae147-4ecc-43a3-840f-f2fd0fe9b56a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:16:29.643698 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.643681    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:16:29.645506 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.645470    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "36dae147-4ecc-43a3-840f-f2fd0fe9b56a" (UID: "36dae147-4ecc-43a3-840f-f2fd0fe9b56a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:29.645628 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.645539    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "36dae147-4ecc-43a3-840f-f2fd0fe9b56a" (UID: "36dae147-4ecc-43a3-840f-f2fd0fe9b56a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:16:29.645628 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.645581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-kube-api-access-8nplw" (OuterVolumeSpecName: "kube-api-access-8nplw") pod "36dae147-4ecc-43a3-840f-f2fd0fe9b56a" (UID: "36dae147-4ecc-43a3-840f-f2fd0fe9b56a"). InnerVolumeSpecName "kube-api-access-8nplw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:16:29.744537 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.744492    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:16:29.744537 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.744531    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:16:29.744537 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.744540    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8nplw\" (UniqueName: \"kubernetes.io/projected/36dae147-4ecc-43a3-840f-f2fd0fe9b56a-kube-api-access-8nplw\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:16:29.831826 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:29.831718    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:30.089602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.089509    2577 generic.go:356] "Generic (PLEG): container finished" podID="36dae147-4ecc-43a3-840f-f2fd0fe9b56a" containerID="bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0" exitCode=0
Sep 10 23:16:30.089602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.089562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96" event={"ID":"36dae147-4ecc-43a3-840f-f2fd0fe9b56a","Type":"ContainerDied","Data":"bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0"}
Sep 10 23:16:30.089602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.089598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96" event={"ID":"36dae147-4ecc-43a3-840f-f2fd0fe9b56a","Type":"ContainerDied","Data":"8d78dc017d8cb0e801d7c510935c709da492d1812ba48262a285f2c4fbe53c3f"}
Sep 10 23:16:30.089910 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.089621    2577 scope.go:117] "RemoveContainer" containerID="bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0"
Sep 10 23:16:30.089910 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.089562    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"
Sep 10 23:16:30.098944 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.098925    2577 scope.go:117] "RemoveContainer" containerID="bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0"
Sep 10 23:16:30.099218 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:16:30.099199    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0\": container with ID starting with bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0 not found: ID does not exist" containerID="bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0"
Sep 10 23:16:30.099273 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.099237    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0"} err="failed to get container status \"bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0\": rpc error: code = NotFound desc = could not find container \"bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0\": container with ID starting with bcd768ffddd2b57f0c39aa6bee3509a563ce05b0ea8bab10262f552a154521b0 not found: ID does not exist"
Sep 10 23:16:30.112174 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.112143    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"]
Sep 10 23:16:30.116057 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.116033    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-8972e2-c594546f-d6m96"]
Sep 10 23:16:30.215351 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:30.215315    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36dae147-4ecc-43a3-840f-f2fd0fe9b56a" path="/var/lib/kubelet/pods/36dae147-4ecc-43a3-840f-f2fd0fe9b56a/volumes"
Sep 10 23:16:32.078887 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:32.078854    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:16:35.994219 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:35.994168    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:36.094421 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:36.094379    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:36.236462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:36.236425    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:36.673886 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:36.673849    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:36.978998 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:36.978903    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:37.075993 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:37.075962    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:16:37.155657 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:37.155626    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:38.835441 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:38.835406    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:38.932535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:38.932502    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:39.035442 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:39.035399    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:45.534096 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:45.534060    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:47.508088 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:47.508042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:47.561006 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:47.560946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.279722 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:48.279690    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.890145 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:48.890107    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:48.991883 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:48.991840    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:49.311618 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:49.311571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:55.517016 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:55.516977    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:16:55.596441 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:16:55.596392    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:06.298333 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.298291    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"]
Sep 10 23:17:06.298773 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.298574    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb" podUID="6bad6c63-0989-4fe5-9d63-1f415df8a5a3" containerName="maas-api" containerID="cri-o://563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09" gracePeriod=30
Sep 10 23:17:06.322528 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.322493    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:06.357465 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.357428    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"]
Sep 10 23:17:06.358029 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.357994    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd" podUID="c347cade-5767-4f3b-96a7-a835e62f5cd5" containerName="payload-processing" containerID="cri-o://01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043" gracePeriod=30
Sep 10 23:17:06.385448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.385416    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:06.441026 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.440988    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:06.543365 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.543322    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:06.603771 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.603735    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:06.637867 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.637844    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:17:06.640795 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.640758    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:17:06.778873 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.778838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpmd4\" (UniqueName: \"kubernetes.io/projected/c347cade-5767-4f3b-96a7-a835e62f5cd5-kube-api-access-wpmd4\") pod \"c347cade-5767-4f3b-96a7-a835e62f5cd5\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") "
Sep 10 23:17:06.778873 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.778879    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c347cade-5767-4f3b-96a7-a835e62f5cd5-plugins-config-volume\") pod \"c347cade-5767-4f3b-96a7-a835e62f5cd5\" (UID: \"c347cade-5767-4f3b-96a7-a835e62f5cd5\") "
Sep 10 23:17:06.779137 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.778909    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-openshift-service-ca\") pod \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") "
Sep 10 23:17:06.779137 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.778936    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-metrics-certs\") pod \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") "
Sep 10 23:17:06.779137 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.778984    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-maas-api-tls\") pod \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") "
Sep 10 23:17:06.779137 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.779036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lgkwx\" (UniqueName: \"kubernetes.io/projected/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-kube-api-access-lgkwx\") pod \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\" (UID: \"6bad6c63-0989-4fe5-9d63-1f415df8a5a3\") "
Sep 10 23:17:06.779362 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.779320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c347cade-5767-4f3b-96a7-a835e62f5cd5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c347cade-5767-4f3b-96a7-a835e62f5cd5" (UID: "c347cade-5767-4f3b-96a7-a835e62f5cd5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:17:06.779498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.779318    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6bad6c63-0989-4fe5-9d63-1f415df8a5a3" (UID: "6bad6c63-0989-4fe5-9d63-1f415df8a5a3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:17:06.781175 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.781151    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6bad6c63-0989-4fe5-9d63-1f415df8a5a3" (UID: "6bad6c63-0989-4fe5-9d63-1f415df8a5a3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:17:06.781282 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.781242    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-kube-api-access-lgkwx" (OuterVolumeSpecName: "kube-api-access-lgkwx") pod "6bad6c63-0989-4fe5-9d63-1f415df8a5a3" (UID: "6bad6c63-0989-4fe5-9d63-1f415df8a5a3"). InnerVolumeSpecName "kube-api-access-lgkwx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:06.781529 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.781505    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6bad6c63-0989-4fe5-9d63-1f415df8a5a3" (UID: "6bad6c63-0989-4fe5-9d63-1f415df8a5a3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:17:06.781601 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.781511    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c347cade-5767-4f3b-96a7-a835e62f5cd5-kube-api-access-wpmd4" (OuterVolumeSpecName: "kube-api-access-wpmd4") pod "c347cade-5767-4f3b-96a7-a835e62f5cd5" (UID: "c347cade-5767-4f3b-96a7-a835e62f5cd5"). InnerVolumeSpecName "kube-api-access-wpmd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:06.880068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.879975    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lgkwx\" (UniqueName: \"kubernetes.io/projected/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-kube-api-access-lgkwx\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:06.880068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.880013    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpmd4\" (UniqueName: \"kubernetes.io/projected/c347cade-5767-4f3b-96a7-a835e62f5cd5-kube-api-access-wpmd4\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:06.880068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.880026    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c347cade-5767-4f3b-96a7-a835e62f5cd5-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:06.880068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.880037    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:06.880068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.880050    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:06.880068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.880062    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6bad6c63-0989-4fe5-9d63-1f415df8a5a3-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:06.919791 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:06.919757    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:07.280046 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.280011    2577 generic.go:356] "Generic (PLEG): container finished" podID="c347cade-5767-4f3b-96a7-a835e62f5cd5" containerID="01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043" exitCode=0
Sep 10 23:17:07.280248 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.280056    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"
Sep 10 23:17:07.280248 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.280090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd" event={"ID":"c347cade-5767-4f3b-96a7-a835e62f5cd5","Type":"ContainerDied","Data":"01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043"}
Sep 10 23:17:07.280248 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.280144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd" event={"ID":"c347cade-5767-4f3b-96a7-a835e62f5cd5","Type":"ContainerDied","Data":"925659423e87214594b8642c799bad70cbc2de007378d70610b3a51c372899f9"}
Sep 10 23:17:07.280248 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.280169    2577 scope.go:117] "RemoveContainer" containerID="01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043"
Sep 10 23:17:07.281846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.281825    2577 generic.go:356] "Generic (PLEG): container finished" podID="6bad6c63-0989-4fe5-9d63-1f415df8a5a3" containerID="563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09" exitCode=0
Sep 10 23:17:07.281955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.281857    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb" event={"ID":"6bad6c63-0989-4fe5-9d63-1f415df8a5a3","Type":"ContainerDied","Data":"563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09"}
Sep 10 23:17:07.281955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.281879    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"
Sep 10 23:17:07.281955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.281892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb" event={"ID":"6bad6c63-0989-4fe5-9d63-1f415df8a5a3","Type":"ContainerDied","Data":"5b26b46221c0d91c10fde2e5955e0aacd8b178883576efc575f2cbf96321f102"}
Sep 10 23:17:07.290843 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.290820    2577 scope.go:117] "RemoveContainer" containerID="01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043"
Sep 10 23:17:07.291157 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:17:07.291138    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043\": container with ID starting with 01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043 not found: ID does not exist" containerID="01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043"
Sep 10 23:17:07.291233 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.291164    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043"} err="failed to get container status \"01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043\": rpc error: code = NotFound desc = could not find container \"01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043\": container with ID starting with 01b0829cb4824a6a174422457960bb270ef83f7bc01869be80e7d486f7287043 not found: ID does not exist"
Sep 10 23:17:07.291233 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.291181    2577 scope.go:117] "RemoveContainer" containerID="563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09"
Sep 10 23:17:07.300603 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.300446    2577 scope.go:117] "RemoveContainer" containerID="563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09"
Sep 10 23:17:07.300865 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:17:07.300749    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09\": container with ID starting with 563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09 not found: ID does not exist" containerID="563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09"
Sep 10 23:17:07.300865 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.300788    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09"} err="failed to get container status \"563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09\": rpc error: code = NotFound desc = could not find container \"563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09\": container with ID starting with 563c3e72125952412efca711cf70ab5a7d43d3ccee3a8414cc4c3b070d474c09 not found: ID does not exist"
Sep 10 23:17:07.303876 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.303852    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"]
Sep 10 23:17:07.311608 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.311583    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-1f90a11e-646f6ddcc-wb2xd"]
Sep 10 23:17:07.325485 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.325460    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"]
Sep 10 23:17:07.329300 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:07.329276    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-1f90a11e-77fcddc7b9-mczjb"]
Sep 10 23:17:08.215087 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:08.215050    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6bad6c63-0989-4fe5-9d63-1f415df8a5a3" path="/var/lib/kubelet/pods/6bad6c63-0989-4fe5-9d63-1f415df8a5a3/volumes"
Sep 10 23:17:08.215471 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:08.215436    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c347cade-5767-4f3b-96a7-a835e62f5cd5" path="/var/lib/kubelet/pods/c347cade-5767-4f3b-96a7-a835e62f5cd5/volumes"
Sep 10 23:17:13.061453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:13.061411    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:13.162196 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:13.162145    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:13.318369 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:13.318275    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:13.642701 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:13.642618    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:13.745491 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:13.745451    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:13.867551 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:13.867508    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:19.868932 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:19.868891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:19.970928 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:19.970887    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:20.068985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:20.068948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:20.201448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:20.201386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:24.166535 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:24.166483    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:24.946197 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:24.946158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:26.337703 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:26.337602    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:26.396615 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:26.396571    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:26.542560 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:26.542520    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"]
Sep 10 23:17:26.565718 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:26.565682    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:27.223963 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:27.223924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:27.470000 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:27.469954    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:27.551112 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:27.551015    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:27.608563 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:27.608522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:27.754500 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:27.754462    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:27.856034 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:27.855946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:27.915317 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:27.915279    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:28.668875 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:28.668833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:30.787817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:30.787759    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:30.889351 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:30.889307    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.103822 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.103597    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"]
Sep 10 23:17:32.104424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104397    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6bad6c63-0989-4fe5-9d63-1f415df8a5a3" containerName="maas-api"
Sep 10 23:17:32.104424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104425    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6bad6c63-0989-4fe5-9d63-1f415df8a5a3" containerName="maas-api"
Sep 10 23:17:32.104612 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104440    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c347cade-5767-4f3b-96a7-a835e62f5cd5" containerName="payload-processing"
Sep 10 23:17:32.104612 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104448    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c347cade-5767-4f3b-96a7-a835e62f5cd5" containerName="payload-processing"
Sep 10 23:17:32.104612 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104458    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36dae147-4ecc-43a3-840f-f2fd0fe9b56a" containerName="maas-api"
Sep 10 23:17:32.104612 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104466    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="36dae147-4ecc-43a3-840f-f2fd0fe9b56a" containerName="maas-api"
Sep 10 23:17:32.104612 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104596    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="36dae147-4ecc-43a3-840f-f2fd0fe9b56a" containerName="maas-api"
Sep 10 23:17:32.104612 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104609    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6bad6c63-0989-4fe5-9d63-1f415df8a5a3" containerName="maas-api"
Sep 10 23:17:32.104944 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.104617    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c347cade-5767-4f3b-96a7-a835e62f5cd5" containerName="payload-processing"
Sep 10 23:17:32.107846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.107829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.111647 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.111606    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-94f305-serving-cert\""
Sep 10 23:17:32.113294 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.113272    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"]
Sep 10 23:17:32.121208 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.121181    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.208003 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.207968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8lkm\" (UniqueName: \"kubernetes.io/projected/a6b8f98b-9f4b-41c8-91c2-845d267b8259-kube-api-access-m8lkm\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.208189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.208141    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.208189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.208169    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.208267 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.208195    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8f98b-9f4b-41c8-91c2-845d267b8259-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.218579 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.218546    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.274627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.274590    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.309578 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.309543    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.309578 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.309579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.309846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.309615    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8f98b-9f4b-41c8-91c2-845d267b8259-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.309846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.309648    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8lkm\" (UniqueName: \"kubernetes.io/projected/a6b8f98b-9f4b-41c8-91c2-845d267b8259-kube-api-access-m8lkm\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.310431 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.310409    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8f98b-9f4b-41c8-91c2-845d267b8259-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.312262 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.312233    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.312408 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.312278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.317515 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.317488    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8lkm\" (UniqueName: \"kubernetes.io/projected/a6b8f98b-9f4b-41c8-91c2-845d267b8259-kube-api-access-m8lkm\") pod \"maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.422583 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.422542    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:32.595756 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.595722    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.697034 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.696854    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:32.698136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:32.698114    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"]
Sep 10 23:17:32.700931 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:17:32.700902    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6b8f98b_9f4b_41c8_91c2_845d267b8259.slice/crio-5ada2e7233d7833a69baf45b7cf447b46722c67980f8decfba6b621262350077 WatchSource:0}: Error finding container 5ada2e7233d7833a69baf45b7cf447b46722c67980f8decfba6b621262350077: Status 404 returned error can't find the container with id 5ada2e7233d7833a69baf45b7cf447b46722c67980f8decfba6b621262350077
Sep 10 23:17:33.003655 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:33.003569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:33.413013 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:33.412976    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p" event={"ID":"a6b8f98b-9f4b-41c8-91c2-845d267b8259","Type":"ContainerStarted","Data":"f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7"}
Sep 10 23:17:33.413422 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:33.413029    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p" event={"ID":"a6b8f98b-9f4b-41c8-91c2-845d267b8259","Type":"ContainerStarted","Data":"5ada2e7233d7833a69baf45b7cf447b46722c67980f8decfba6b621262350077"}
Sep 10 23:17:33.413422 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:33.413059    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:33.432169 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:33.432110    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p" podStartSLOduration=0.847929511 podStartE2EDuration="1.432089082s" totalImagesPullingTime="584.159571ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:17:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:17:33.430279803 +0000 UTC m=+1169.866334576" watchObservedRunningTime="2026-09-10 23:17:33.432089082 +0000 UTC m=+1169.868143854"
Sep 10 23:17:38.900526 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:38.900478    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:39.423162 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.423134    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:17:39.465233 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.465198    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"]
Sep 10 23:17:39.465473 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.465444    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw" podUID="4e9a3890-94f6-4b89-8b4b-805a9585cf12" containerName="maas-api" containerID="cri-o://b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4" gracePeriod=30
Sep 10 23:17:39.476108 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.476074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:39.567627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.567594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:39.624597 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.624557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:39.681203 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.681109    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:39.718791 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.718751    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:17:39.886106 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.886065    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-922dr\" (UniqueName: \"kubernetes.io/projected/4e9a3890-94f6-4b89-8b4b-805a9585cf12-kube-api-access-922dr\") pod \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") "
Sep 10 23:17:39.886301 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.886138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls\") pod \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") "
Sep 10 23:17:39.886301 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.886236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9a3890-94f6-4b89-8b4b-805a9585cf12-openshift-service-ca\") pod \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") "
Sep 10 23:17:39.886301 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.886273    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-metrics-certs\") pod \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\" (UID: \"4e9a3890-94f6-4b89-8b4b-805a9585cf12\") "
Sep 10 23:17:39.886595 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.886563    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e9a3890-94f6-4b89-8b4b-805a9585cf12-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4e9a3890-94f6-4b89-8b4b-805a9585cf12" (UID: "4e9a3890-94f6-4b89-8b4b-805a9585cf12"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:17:39.888411 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.888378    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e9a3890-94f6-4b89-8b4b-805a9585cf12-kube-api-access-922dr" (OuterVolumeSpecName: "kube-api-access-922dr") pod "4e9a3890-94f6-4b89-8b4b-805a9585cf12" (UID: "4e9a3890-94f6-4b89-8b4b-805a9585cf12"). InnerVolumeSpecName "kube-api-access-922dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:39.888553 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.888487    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4e9a3890-94f6-4b89-8b4b-805a9585cf12" (UID: "4e9a3890-94f6-4b89-8b4b-805a9585cf12"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:17:39.888553 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.888516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "4e9a3890-94f6-4b89-8b4b-805a9585cf12" (UID: "4e9a3890-94f6-4b89-8b4b-805a9585cf12"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:17:39.987227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.987151    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e9a3890-94f6-4b89-8b4b-805a9585cf12-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:39.987227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.987180    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:39.987227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.987190    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-922dr\" (UniqueName: \"kubernetes.io/projected/4e9a3890-94f6-4b89-8b4b-805a9585cf12-kube-api-access-922dr\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:39.987227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:39.987199    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4e9a3890-94f6-4b89-8b4b-805a9585cf12-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:40.009849 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.009818    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:40.449909 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.449874    2577 generic.go:356] "Generic (PLEG): container finished" podID="4e9a3890-94f6-4b89-8b4b-805a9585cf12" containerID="b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4" exitCode=0
Sep 10 23:17:40.450123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.449924    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"
Sep 10 23:17:40.450123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.449959    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw" event={"ID":"4e9a3890-94f6-4b89-8b4b-805a9585cf12","Type":"ContainerDied","Data":"b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4"}
Sep 10 23:17:40.450123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.450008    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw" event={"ID":"4e9a3890-94f6-4b89-8b4b-805a9585cf12","Type":"ContainerDied","Data":"3cf5d299f00db3e1ae5814e57b4c54bc75935d1606e00be67de4ebb25cccdec0"}
Sep 10 23:17:40.450123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.450034    2577 scope.go:117] "RemoveContainer" containerID="b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4"
Sep 10 23:17:40.458863 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.458841    2577 scope.go:117] "RemoveContainer" containerID="b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4"
Sep 10 23:17:40.459184 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:17:40.459164    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4\": container with ID starting with b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4 not found: ID does not exist" containerID="b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4"
Sep 10 23:17:40.459233 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.459195    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4"} err="failed to get container status \"b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4\": rpc error: code = NotFound desc = could not find container \"b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4\": container with ID starting with b26c804f0a32b2e126afc1dcfc8d8692c4a036fc1327b1b7af0e4ff62955eda4 not found: ID does not exist"
Sep 10 23:17:40.467284 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.467257    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"]
Sep 10 23:17:40.471446 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:40.471418    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-28dbae-5994d47f87-k2pjw"]
Sep 10 23:17:41.562627 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.562586    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg" podUID="afcc8f99-4270-4dad-94b6-2fc713703918" containerName="main" containerID="cri-o://e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0" gracePeriod=45
Sep 10 23:17:41.806809 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.806771    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:17:41.905267 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905235    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-tmp-dir\") pod \"afcc8f99-4270-4dad-94b6-2fc713703918\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") "
Sep 10 23:17:41.905462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905283    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc8f99-4270-4dad-94b6-2fc713703918-tls-certs\") pod \"afcc8f99-4270-4dad-94b6-2fc713703918\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") "
Sep 10 23:17:41.905462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905308    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-model-cache\") pod \"afcc8f99-4270-4dad-94b6-2fc713703918\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") "
Sep 10 23:17:41.905462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-home\") pod \"afcc8f99-4270-4dad-94b6-2fc713703918\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") "
Sep 10 23:17:41.905462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905370    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pprq7\" (UniqueName: \"kubernetes.io/projected/afcc8f99-4270-4dad-94b6-2fc713703918-kube-api-access-pprq7\") pod \"afcc8f99-4270-4dad-94b6-2fc713703918\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") "
Sep 10 23:17:41.905672 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-dshm\") pod \"afcc8f99-4270-4dad-94b6-2fc713703918\" (UID: \"afcc8f99-4270-4dad-94b6-2fc713703918\") "
Sep 10 23:17:41.905672 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905564    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "afcc8f99-4270-4dad-94b6-2fc713703918" (UID: "afcc8f99-4270-4dad-94b6-2fc713703918"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:41.905672 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905572    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-model-cache" (OuterVolumeSpecName: "model-cache") pod "afcc8f99-4270-4dad-94b6-2fc713703918" (UID: "afcc8f99-4270-4dad-94b6-2fc713703918"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:41.905672 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-home" (OuterVolumeSpecName: "home") pod "afcc8f99-4270-4dad-94b6-2fc713703918" (UID: "afcc8f99-4270-4dad-94b6-2fc713703918"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:41.905894 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905773    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-tmp-dir\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:41.905894 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905818    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-model-cache\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:41.905894 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.905832    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-home\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:41.907603 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.907580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/afcc8f99-4270-4dad-94b6-2fc713703918-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "afcc8f99-4270-4dad-94b6-2fc713703918" (UID: "afcc8f99-4270-4dad-94b6-2fc713703918"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:17:41.907603 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.907590    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-dshm" (OuterVolumeSpecName: "dshm") pod "afcc8f99-4270-4dad-94b6-2fc713703918" (UID: "afcc8f99-4270-4dad-94b6-2fc713703918"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 23:17:41.907802 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:41.907594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/afcc8f99-4270-4dad-94b6-2fc713703918-kube-api-access-pprq7" (OuterVolumeSpecName: "kube-api-access-pprq7") pod "afcc8f99-4270-4dad-94b6-2fc713703918" (UID: "afcc8f99-4270-4dad-94b6-2fc713703918"). InnerVolumeSpecName "kube-api-access-pprq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:42.006566 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.006485    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/afcc8f99-4270-4dad-94b6-2fc713703918-dshm\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:42.006566 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.006514    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/afcc8f99-4270-4dad-94b6-2fc713703918-tls-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:42.006566 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.006526    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pprq7\" (UniqueName: \"kubernetes.io/projected/afcc8f99-4270-4dad-94b6-2fc713703918-kube-api-access-pprq7\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:42.219933 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.219901    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e9a3890-94f6-4b89-8b4b-805a9585cf12" path="/var/lib/kubelet/pods/4e9a3890-94f6-4b89-8b4b-805a9585cf12/volumes"
Sep 10 23:17:42.462570 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.462532    2577 generic.go:356] "Generic (PLEG): container finished" podID="afcc8f99-4270-4dad-94b6-2fc713703918" containerID="e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0" exitCode=0
Sep 10 23:17:42.462734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.462594    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"
Sep 10 23:17:42.462734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.462590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg" event={"ID":"afcc8f99-4270-4dad-94b6-2fc713703918","Type":"ContainerDied","Data":"e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0"}
Sep 10 23:17:42.462734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.462706    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg" event={"ID":"afcc8f99-4270-4dad-94b6-2fc713703918","Type":"ContainerDied","Data":"235ecb67402e161d58956ec201ad758f665be137f5e65b4821ed25bc61b3fdb4"}
Sep 10 23:17:42.462734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.462727    2577 scope.go:117] "RemoveContainer" containerID="e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0"
Sep 10 23:17:42.471405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.471382    2577 scope.go:117] "RemoveContainer" containerID="e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0"
Sep 10 23:17:42.471682 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:17:42.471660    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0\": container with ID starting with e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0 not found: ID does not exist" containerID="e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0"
Sep 10 23:17:42.471759 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.471702    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0"} err="failed to get container status \"e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0\": rpc error: code = NotFound desc = could not find container \"e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0\": container with ID starting with e2e682d8f277c95d64d9441eb5b53c88a9b9d79b9b853ab88a76d31ef98574f0 not found: ID does not exist"
Sep 10 23:17:42.485313 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.485277    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"]
Sep 10 23:17:42.489182 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:42.489145    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-28dbae/test-model-28dbae-kserve-5fc9768ffc-qh4gg"]
Sep 10 23:17:44.215044 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:44.215008    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afcc8f99-4270-4dad-94b6-2fc713703918" path="/var/lib/kubelet/pods/afcc8f99-4270-4dad-94b6-2fc713703918/volumes"
Sep 10 23:17:46.246556 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:46.246523    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:46.302634 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:46.302594    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:46.494973 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:46.494934    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:46.988916 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:46.988882    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:52.981146 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:52.981106    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:54.632275 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:54.632230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:55.746467 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:55.746427    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:56.083114 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:56.083023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:56.408592 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:56.408560    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:56.620635 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:56.620593    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:57.510285 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.510250    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"]
Sep 10 23:17:57.510713 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.510470    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x" podUID="2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" containerName="maas-api" containerID="cri-o://4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8" gracePeriod=30
Sep 10 23:17:57.541371 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.541330    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:57.570409 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.570378    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"]
Sep 10 23:17:57.570625 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.570587    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp" podUID="8fc155a6-61a5-4113-ae8f-b080310aea50" containerName="payload-processing" containerID="cri-o://ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd" gracePeriod=30
Sep 10 23:17:57.639509 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.639467    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:57.738700 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.738649    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:57.836258 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.836235    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:17:57.839260 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.839228    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:57.846791 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.846757    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:17:57.938941 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.938911    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-openshift-service-ca\") pod \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") "
Sep 10 23:17:57.939150 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.938950    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-metrics-certs\") pod \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") "
Sep 10 23:17:57.939150 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.939015    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8stx5\" (UniqueName: \"kubernetes.io/projected/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-kube-api-access-8stx5\") pod \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") "
Sep 10 23:17:57.939150 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.939046    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc155a6-61a5-4113-ae8f-b080310aea50-plugins-config-volume\") pod \"8fc155a6-61a5-4113-ae8f-b080310aea50\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") "
Sep 10 23:17:57.939150 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.939078    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-64x95\" (UniqueName: \"kubernetes.io/projected/8fc155a6-61a5-4113-ae8f-b080310aea50-kube-api-access-64x95\") pod \"8fc155a6-61a5-4113-ae8f-b080310aea50\" (UID: \"8fc155a6-61a5-4113-ae8f-b080310aea50\") "
Sep 10 23:17:57.939150 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.939114    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls\") pod \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\" (UID: \"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0\") "
Sep 10 23:17:57.939405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.939322    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" (UID: "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:17:57.939405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.939392    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fc155a6-61a5-4113-ae8f-b080310aea50-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8fc155a6-61a5-4113-ae8f-b080310aea50" (UID: "8fc155a6-61a5-4113-ae8f-b080310aea50"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:17:57.939516 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.939417    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:57.941429 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.941397    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-kube-api-access-8stx5" (OuterVolumeSpecName: "kube-api-access-8stx5") pod "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" (UID: "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0"). InnerVolumeSpecName "kube-api-access-8stx5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:57.941429 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.941394    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fc155a6-61a5-4113-ae8f-b080310aea50-kube-api-access-64x95" (OuterVolumeSpecName: "kube-api-access-64x95") pod "8fc155a6-61a5-4113-ae8f-b080310aea50" (UID: "8fc155a6-61a5-4113-ae8f-b080310aea50"). InnerVolumeSpecName "kube-api-access-64x95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:17:57.941429 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.941399    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" (UID: "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:17:57.941615 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:57.941463    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" (UID: "2891bfb3-de47-4c10-8c39-2b1d1be4e0a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:17:58.040271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.040187    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:58.040271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.040216    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:58.040271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.040226    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8stx5\" (UniqueName: \"kubernetes.io/projected/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0-kube-api-access-8stx5\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:58.040271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.040235    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8fc155a6-61a5-4113-ae8f-b080310aea50-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:58.040271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.040244    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64x95\" (UniqueName: \"kubernetes.io/projected/8fc155a6-61a5-4113-ae8f-b080310aea50-kube-api-access-64x95\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:17:58.064831 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.064790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:58.554315 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.554279    2577 generic.go:356] "Generic (PLEG): container finished" podID="8fc155a6-61a5-4113-ae8f-b080310aea50" containerID="ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd" exitCode=0
Sep 10 23:17:58.554804 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.554347    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"
Sep 10 23:17:58.554804 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.554365    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp" event={"ID":"8fc155a6-61a5-4113-ae8f-b080310aea50","Type":"ContainerDied","Data":"ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd"}
Sep 10 23:17:58.554804 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.554467    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp" event={"ID":"8fc155a6-61a5-4113-ae8f-b080310aea50","Type":"ContainerDied","Data":"15d35ab56acaae92a23a13d8ecf30055131f40f41d1719595e1593c43fc6af1d"}
Sep 10 23:17:58.554804 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.554489    2577 scope.go:117] "RemoveContainer" containerID="ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd"
Sep 10 23:17:58.556021 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.556000    2577 generic.go:356] "Generic (PLEG): container finished" podID="2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" containerID="4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8" exitCode=0
Sep 10 23:17:58.556101 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.556067    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"
Sep 10 23:17:58.556101 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.556082    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x" event={"ID":"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0","Type":"ContainerDied","Data":"4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8"}
Sep 10 23:17:58.556186 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.556132    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x" event={"ID":"2891bfb3-de47-4c10-8c39-2b1d1be4e0a0","Type":"ContainerDied","Data":"55a48b3cdf74d3377acdeec864dc4a8c32c68cd76d2e90b72fb53261c0cdbf1e"}
Sep 10 23:17:58.563189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.563159    2577 scope.go:117] "RemoveContainer" containerID="ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd"
Sep 10 23:17:58.563458 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:17:58.563433    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd\": container with ID starting with ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd not found: ID does not exist" containerID="ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd"
Sep 10 23:17:58.563549 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.563470    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd"} err="failed to get container status \"ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd\": rpc error: code = NotFound desc = could not find container \"ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd\": container with ID starting with ea6a59a42239b846bee6e3368ab09c0ae636bd95936eb8537885ed7ae0ff24fd not found: ID does not exist"
Sep 10 23:17:58.563549 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.563493    2577 scope.go:117] "RemoveContainer" containerID="4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8"
Sep 10 23:17:58.571439 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.571336    2577 scope.go:117] "RemoveContainer" containerID="4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8"
Sep 10 23:17:58.571824 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:17:58.571803    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8\": container with ID starting with 4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8 not found: ID does not exist" containerID="4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8"
Sep 10 23:17:58.571903 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.571830    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8"} err="failed to get container status \"4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8\": rpc error: code = NotFound desc = could not find container \"4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8\": container with ID starting with 4afbdfe685e5df4aceee7b5356279299372f8827d7c1549936dc099b6db442a8 not found: ID does not exist"
Sep 10 23:17:58.573302 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.573282    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"]
Sep 10 23:17:58.577507 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.577486    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-1f1f85-685664577c-pv9tp"]
Sep 10 23:17:58.587123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.587099    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"]
Sep 10 23:17:58.591193 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:58.591159    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-1f1f85-b97bbf4fc-mvr7x"]
Sep 10 23:17:59.038904 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:59.038853    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:17:59.138575 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:17:59.138536    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.171763 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.171730    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"]
Sep 10 23:18:00.172306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172248    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e9a3890-94f6-4b89-8b4b-805a9585cf12" containerName="maas-api"
Sep 10 23:18:00.172306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172268    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e9a3890-94f6-4b89-8b4b-805a9585cf12" containerName="maas-api"
Sep 10 23:18:00.172306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172281    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8fc155a6-61a5-4113-ae8f-b080310aea50" containerName="payload-processing"
Sep 10 23:18:00.172306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172290    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8fc155a6-61a5-4113-ae8f-b080310aea50" containerName="payload-processing"
Sep 10 23:18:00.172306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172303    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" containerName="maas-api"
Sep 10 23:18:00.172306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172311    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" containerName="maas-api"
Sep 10 23:18:00.172691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172320    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afcc8f99-4270-4dad-94b6-2fc713703918" containerName="main"
Sep 10 23:18:00.172691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172328    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="afcc8f99-4270-4dad-94b6-2fc713703918" containerName="main"
Sep 10 23:18:00.172691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172446    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" containerName="maas-api"
Sep 10 23:18:00.172691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172459    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e9a3890-94f6-4b89-8b4b-805a9585cf12" containerName="maas-api"
Sep 10 23:18:00.172691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172469    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8fc155a6-61a5-4113-ae8f-b080310aea50" containerName="payload-processing"
Sep 10 23:18:00.172691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.172477    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="afcc8f99-4270-4dad-94b6-2fc713703918" containerName="main"
Sep 10 23:18:00.178014 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.177988    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.180245 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.180219    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-063e1d-serving-cert\""
Sep 10 23:18:00.185149 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.185119    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"]
Sep 10 23:18:00.202561 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.202530    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.214912 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.214882    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2891bfb3-de47-4c10-8c39-2b1d1be4e0a0" path="/var/lib/kubelet/pods/2891bfb3-de47-4c10-8c39-2b1d1be4e0a0/volumes"
Sep 10 23:18:00.215291 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.215276    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8fc155a6-61a5-4113-ae8f-b080310aea50" path="/var/lib/kubelet/pods/8fc155a6-61a5-4113-ae8f-b080310aea50/volumes"
Sep 10 23:18:00.260435 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.260396    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af934bd-1d1b-4b3d-beeb-50322b80155d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.260646 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.260459    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hl5c5\" (UniqueName: \"kubernetes.io/projected/1af934bd-1d1b-4b3d-beeb-50322b80155d-kube-api-access-hl5c5\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.260646 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.260588    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.260646 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.260629    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.303423 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.303385    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.361313 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.361255    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.361530 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.361346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af934bd-1d1b-4b3d-beeb-50322b80155d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.361530 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.361384    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hl5c5\" (UniqueName: \"kubernetes.io/projected/1af934bd-1d1b-4b3d-beeb-50322b80155d-kube-api-access-hl5c5\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.361530 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.361465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.362015 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.361990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af934bd-1d1b-4b3d-beeb-50322b80155d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.363846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.363823    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.363917 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.363844    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.370029 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.369994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hl5c5\" (UniqueName: \"kubernetes.io/projected/1af934bd-1d1b-4b3d-beeb-50322b80155d-kube-api-access-hl5c5\") pod \"maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.405794 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.405753    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.492885 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.492805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:00.680201 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.680170    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.769976 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.769889    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"]
Sep 10 23:18:00.773136 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:00.773096    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1af934bd_1d1b_4b3d_beeb_50322b80155d.slice/crio-e6c6c7dd833c8a208f9cc09c98be7ffff6741b3c25092d18922f93dabac46fb4 WatchSource:0}: Error finding container e6c6c7dd833c8a208f9cc09c98be7ffff6741b3c25092d18922f93dabac46fb4: Status 404 returned error can't find the container with id e6c6c7dd833c8a208f9cc09c98be7ffff6741b3c25092d18922f93dabac46fb4
Sep 10 23:18:00.773136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.773129    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:00.787357 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.787332    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"]
Sep 10 23:18:00.791890 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.791870    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:00.794002 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.793983    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-063e1d\""
Sep 10 23:18:00.794108 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.794051    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-063e1d-dockercfg-fh96c\""
Sep 10 23:18:00.800835 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.800815    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"]
Sep 10 23:18:00.867141 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.867097    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln2bl\" (UniqueName: \"kubernetes.io/projected/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-kube-api-access-ln2bl\") pod \"payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:00.867334 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.867159    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:00.968249 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.968205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln2bl\" (UniqueName: \"kubernetes.io/projected/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-kube-api-access-ln2bl\") pod \"payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:00.968419 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.968369    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:00.969300 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.969277    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:00.977090 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:00.977069    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln2bl\" (UniqueName: \"kubernetes.io/projected/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-kube-api-access-ln2bl\") pod \"payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:01.103392 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.103297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:01.375405 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.375200    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"]
Sep 10 23:18:01.378090 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:01.378055    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dcb74d8_d774_4c31_bd2b_2c6a9e197aa7.slice/crio-7b2f352bb9e408983fd68c0a27fdde4473b5fde6c283f09b1bfb9ab1536bcf05 WatchSource:0}: Error finding container 7b2f352bb9e408983fd68c0a27fdde4473b5fde6c283f09b1bfb9ab1536bcf05: Status 404 returned error can't find the container with id 7b2f352bb9e408983fd68c0a27fdde4473b5fde6c283f09b1bfb9ab1536bcf05
Sep 10 23:18:01.573133 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.573098    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4" event={"ID":"1af934bd-1d1b-4b3d-beeb-50322b80155d","Type":"ContainerStarted","Data":"b806be658d8d778b2469fca91794d08f27d7ae1dce72713bc9d0cd59611f0748"}
Sep 10 23:18:01.573327 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.573164    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:01.573327 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.573192    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4" event={"ID":"1af934bd-1d1b-4b3d-beeb-50322b80155d","Type":"ContainerStarted","Data":"e6c6c7dd833c8a208f9cc09c98be7ffff6741b3c25092d18922f93dabac46fb4"}
Sep 10 23:18:01.574709 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.574672    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q" event={"ID":"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7","Type":"ContainerStarted","Data":"df9b1b92cf5521f820181f7208ae0335797dd3d89e2b0e10e1a3a358d6e44743"}
Sep 10 23:18:01.574869 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.574714    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q" event={"ID":"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7","Type":"ContainerStarted","Data":"7b2f352bb9e408983fd68c0a27fdde4473b5fde6c283f09b1bfb9ab1536bcf05"}
Sep 10 23:18:01.574869 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.574765    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:01.590721 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.590665    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4" podStartSLOduration=1.100904176 podStartE2EDuration="1.590644121s" totalImagesPullingTime="489.739945ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:01.588404478 +0000 UTC m=+1198.024459248" watchObservedRunningTime="2026-09-10 23:18:01.590644121 +0000 UTC m=+1198.026698894"
Sep 10 23:18:01.606503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:01.606453    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q" podStartSLOduration=1.606435744 podStartE2EDuration="1.606435744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:01.605427384 +0000 UTC m=+1198.041482182" watchObservedRunningTime="2026-09-10 23:18:01.606435744 +0000 UTC m=+1198.042490516"
Sep 10 23:18:04.161178 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:04.161135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:04.261943 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:04.261907    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:04.413418 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:04.413334    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:04.953456 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:04.953420    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:05.233624 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:05.233543    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:05.376954 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:05.376915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:07.585424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:07.585391    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:18:08.210332 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:08.210288    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:08.302607 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:08.302565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:09.468760 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.468369    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"]
Sep 10 23:18:09.473909 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.473880    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.476152 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.476128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-2da572-serving-cert\""
Sep 10 23:18:09.484985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.484957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"]
Sep 10 23:18:09.505961 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.505927    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:09.540763 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.540724    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.540763 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.540765    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.541045 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.540815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldnqd\" (UniqueName: \"kubernetes.io/projected/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-kube-api-access-ldnqd\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.541045 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.540868    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.606523 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.606487    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:09.642043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.641997    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.642043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.642042    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.642313 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.642078    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldnqd\" (UniqueName: \"kubernetes.io/projected/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-kube-api-access-ldnqd\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.642313 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.642125    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.642313 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:09.642208    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-serving-cert: secret "maas-api-e2e-shared-a-w3-2da572-serving-cert" not found
Sep 10 23:18:09.642313 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:09.642281    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls podName:28f9b8b7-b116-4b4d-9416-fc7b597b5ff0 nodeName:}" failed. No retries permitted until 2026-09-10 23:18:10.14226226 +0000 UTC m=+1206.578317013 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" (UID: "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0") : secret "maas-api-e2e-shared-a-w3-2da572-serving-cert" not found
Sep 10 23:18:09.642749 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.642732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.644611 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.644590    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:09.652687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:09.652659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldnqd\" (UniqueName: \"kubernetes.io/projected/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-kube-api-access-ldnqd\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:10.068820 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.068749    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:10.147468 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.147430    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:10.147645 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:10.147582    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-serving-cert: secret "maas-api-e2e-shared-a-w3-2da572-serving-cert" not found
Sep 10 23:18:10.147699 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:10.147661    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls podName:28f9b8b7-b116-4b4d-9416-fc7b597b5ff0 nodeName:}" failed. No retries permitted until 2026-09-10 23:18:11.14764506 +0000 UTC m=+1207.583699812 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls") pod "maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" (UID: "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0") : secret "maas-api-e2e-shared-a-w3-2da572-serving-cert" not found
Sep 10 23:18:10.173386 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.173348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:10.182572 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.182542    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"]
Sep 10 23:18:10.186052 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.186036    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.188178 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.188152    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-2da572\""
Sep 10 23:18:10.188295 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.188264    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-2da572-dockercfg-76l4w\""
Sep 10 23:18:10.195651 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.195630    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"]
Sep 10 23:18:10.248775 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.248735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c666fb3-a182-4034-af8c-149c7a16f8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.248986 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.248913    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68gzr\" (UniqueName: \"kubernetes.io/projected/9c666fb3-a182-4034-af8c-149c7a16f8b5-kube-api-access-68gzr\") pod \"payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.268160 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.268131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:10.350011 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.349923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c666fb3-a182-4034-af8c-149c7a16f8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.350214 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.350179    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68gzr\" (UniqueName: \"kubernetes.io/projected/9c666fb3-a182-4034-af8c-149c7a16f8b5-kube-api-access-68gzr\") pod \"payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.350747 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.350724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c666fb3-a182-4034-af8c-149c7a16f8b5-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.358275 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.358244    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-68gzr\" (UniqueName: \"kubernetes.io/projected/9c666fb3-a182-4034-af8c-149c7a16f8b5-kube-api-access-68gzr\") pod \"payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.499801 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.499742    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:10.771703 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:10.771656    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"]
Sep 10 23:18:10.774201 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:10.774171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c666fb3_a182_4034_af8c_149c7a16f8b5.slice/crio-1096ca5cc5df74faa483e1152052147d7e67c3d01bd0d2cdd424da25ef41e82f WatchSource:0}: Error finding container 1096ca5cc5df74faa483e1152052147d7e67c3d01bd0d2cdd424da25ef41e82f: Status 404 returned error can't find the container with id 1096ca5cc5df74faa483e1152052147d7e67c3d01bd0d2cdd424da25ef41e82f
Sep 10 23:18:11.090075 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.089970    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:11.156908 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.156867    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:11.159469 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.159440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:11.289625 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.289583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:11.559889 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.559862    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"]
Sep 10 23:18:11.562841 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:11.562810    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28f9b8b7_b116_4b4d_9416_fc7b597b5ff0.slice/crio-b232710ba020c2b0a1c866911083e695d3f92b8d69eec86f45f38f4d9b0cf922 WatchSource:0}: Error finding container b232710ba020c2b0a1c866911083e695d3f92b8d69eec86f45f38f4d9b0cf922: Status 404 returned error can't find the container with id b232710ba020c2b0a1c866911083e695d3f92b8d69eec86f45f38f4d9b0cf922
Sep 10 23:18:11.623562 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.623527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" event={"ID":"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0","Type":"ContainerStarted","Data":"b232710ba020c2b0a1c866911083e695d3f92b8d69eec86f45f38f4d9b0cf922"}
Sep 10 23:18:11.625043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.625015    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn" event={"ID":"9c666fb3-a182-4034-af8c-149c7a16f8b5","Type":"ContainerStarted","Data":"9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a"}
Sep 10 23:18:11.625171 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.625057    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn" event={"ID":"9c666fb3-a182-4034-af8c-149c7a16f8b5","Type":"ContainerStarted","Data":"1096ca5cc5df74faa483e1152052147d7e67c3d01bd0d2cdd424da25ef41e82f"}
Sep 10 23:18:11.625260 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.625202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:11.644670 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:11.644627    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn" podStartSLOduration=1.644612696 podStartE2EDuration="1.644612696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:11.64262691 +0000 UTC m=+1208.078681686" watchObservedRunningTime="2026-09-10 23:18:11.644612696 +0000 UTC m=+1208.080667471"
Sep 10 23:18:12.582285 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:12.582250    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:18:12.631503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:12.631466    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" event={"ID":"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0","Type":"ContainerStarted","Data":"216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763"}
Sep 10 23:18:12.631723 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:12.631702    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:12.648881 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:12.648827    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" podStartSLOduration=3.360756618 podStartE2EDuration="3.648810573s" totalImagesPullingTime="288.053955ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:12.648097526 +0000 UTC m=+1209.084152300" watchObservedRunningTime="2026-09-10 23:18:12.648810573 +0000 UTC m=+1209.084865362"
Sep 10 23:18:18.641967 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:18.641931    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:18:22.129144 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:22.129102    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:22.634141 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:22.634104    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:18:24.006160 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:24.006120    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:30.488254 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:30.488218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.041824 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:33.041761    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.392319 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:33.392223    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.691975 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:33.691938    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:33.987904 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:33.987811    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:35.456247 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:35.456208    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:35.764191 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:35.764096    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:36.081665 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:36.081557    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:36.384464 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:36.384365    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:36.483620 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:36.483585    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:37.590971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.590928    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"]
Sep 10 23:18:37.594513 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.594490    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.596548 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.596514    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-4b268f-serving-cert\""
Sep 10 23:18:37.603578 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.603552    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"]
Sep 10 23:18:37.673166 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.673135    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:37.703964 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.703934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.703964 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.703985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.704166 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.704008    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjkpg\" (UniqueName: \"kubernetes.io/projected/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-kube-api-access-jjkpg\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.704166 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.704052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.773547 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.773513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:37.804908 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.804881    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.805031 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.804933    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.805031 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.804952    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjkpg\" (UniqueName: \"kubernetes.io/projected/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-kube-api-access-jjkpg\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.805031 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.804974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.805192 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:37.805094    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-serving-cert: secret "maas-api-e2e-shared-b-w3-4b268f-serving-cert" not found
Sep 10 23:18:37.805192 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:37.805182    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls podName:cf228744-5cd6-4bf6-bf78-d67a476f2dd2 nodeName:}" failed. No retries permitted until 2026-09-10 23:18:38.305158623 +0000 UTC m=+1234.741213390 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls") pod "maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" (UID: "cf228744-5cd6-4bf6-bf78-d67a476f2dd2") : secret "maas-api-e2e-shared-b-w3-4b268f-serving-cert" not found
Sep 10 23:18:37.805665 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.805649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.807389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.807371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:37.815821 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:37.815796    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjkpg\" (UniqueName: \"kubernetes.io/projected/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-kube-api-access-jjkpg\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:38.141246 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.141214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:38.226661 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.226626    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"]
Sep 10 23:18:38.228959 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.228930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:38.230410 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.230394    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.232361 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.232341    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-4b268f\""
Sep 10 23:18:38.232463 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.232424    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-4b268f-dockercfg-5xwx7\""
Sep 10 23:18:38.244896 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.244873    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"]
Sep 10 23:18:38.276156 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.276125    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"]
Sep 10 23:18:38.286007 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.285967    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.287757 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.287726    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"]
Sep 10 23:18:38.310238 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.310203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:38.310398 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:38.310355    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-serving-cert: secret "maas-api-e2e-shared-b-w3-4b268f-serving-cert" not found
Sep 10 23:18:38.310470 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:18:38.310438    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls podName:cf228744-5cd6-4bf6-bf78-d67a476f2dd2 nodeName:}" failed. No retries permitted until 2026-09-10 23:18:39.310421268 +0000 UTC m=+1235.746476021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls") pod "maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" (UID: "cf228744-5cd6-4bf6-bf78-d67a476f2dd2") : secret "maas-api-e2e-shared-b-w3-4b268f-serving-cert" not found
Sep 10 23:18:38.402589 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.402552    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:38.411813 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.411753    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdkwc\" (UniqueName: \"kubernetes.io/projected/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-kube-api-access-qdkwc\") pod \"payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.412025 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.411828    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b32c4-c119-482c-9f62-dfc4f2ce428f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.412025 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.411996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.412180 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.412052    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5sfq\" (UniqueName: \"kubernetes.io/projected/e78b32c4-c119-482c-9f62-dfc4f2ce428f-kube-api-access-t5sfq\") pod \"payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.502015 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.501972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:38.512813 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.512762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.512990 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.512907    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5sfq\" (UniqueName: \"kubernetes.io/projected/e78b32c4-c119-482c-9f62-dfc4f2ce428f-kube-api-access-t5sfq\") pod \"payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.512990 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.512972    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdkwc\" (UniqueName: \"kubernetes.io/projected/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-kube-api-access-qdkwc\") pod \"payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.513106 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.513006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b32c4-c119-482c-9f62-dfc4f2ce428f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.513599 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.513577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.513760 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.513742    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b32c4-c119-482c-9f62-dfc4f2ce428f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.520947 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.520924    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5sfq\" (UniqueName: \"kubernetes.io/projected/e78b32c4-c119-482c-9f62-dfc4f2ce428f-kube-api-access-t5sfq\") pod \"payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.521046 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.520997    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdkwc\" (UniqueName: \"kubernetes.io/projected/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-kube-api-access-qdkwc\") pod \"payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.540610 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.540583    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:38.599370 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.599340    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:38.811980 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.811957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"]
Sep 10 23:18:38.814402 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:38.814374    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7047ce95_0fa3_45a1_ab15_6aecbf49c32f.slice/crio-2b90e71f9155487ea9de868bfa7cfeac18023f0b8efe1d55653779928d4ce2f9 WatchSource:0}: Error finding container 2b90e71f9155487ea9de868bfa7cfeac18023f0b8efe1d55653779928d4ce2f9: Status 404 returned error can't find the container with id 2b90e71f9155487ea9de868bfa7cfeac18023f0b8efe1d55653779928d4ce2f9
Sep 10 23:18:38.875568 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:38.875548    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"]
Sep 10 23:18:38.878091 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:38.878060    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78b32c4_c119_482c_9f62_dfc4f2ce428f.slice/crio-e552d68666550510f270d8d9473e09584b3d2bb5e6c38e726f86ec8866d47740 WatchSource:0}: Error finding container e552d68666550510f270d8d9473e09584b3d2bb5e6c38e726f86ec8866d47740: Status 404 returned error can't find the container with id e552d68666550510f270d8d9473e09584b3d2bb5e6c38e726f86ec8866d47740
Sep 10 23:18:39.320841 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.320719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:39.323170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.323150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:39.405206 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.405175    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:39.682866 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.682833    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:39.690959 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.690934    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"]
Sep 10 23:18:39.693987 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:39.693960    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf228744_5cd6_4bf6_bf78_d67a476f2dd2.slice/crio-380ee6e895f524b9cb72a2d9ca5cd421c972e09b52875a327c9ee3071fb9d522 WatchSource:0}: Error finding container 380ee6e895f524b9cb72a2d9ca5cd421c972e09b52875a327c9ee3071fb9d522: Status 404 returned error can't find the container with id 380ee6e895f524b9cb72a2d9ca5cd421c972e09b52875a327c9ee3071fb9d522
Sep 10 23:18:39.764151 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.764109    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" event={"ID":"cf228744-5cd6-4bf6-bf78-d67a476f2dd2","Type":"ContainerStarted","Data":"380ee6e895f524b9cb72a2d9ca5cd421c972e09b52875a327c9ee3071fb9d522"}
Sep 10 23:18:39.765861 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.765825    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87" event={"ID":"7047ce95-0fa3-45a1-ab15-6aecbf49c32f","Type":"ContainerStarted","Data":"8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306"}
Sep 10 23:18:39.766011 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.765888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87" event={"ID":"7047ce95-0fa3-45a1-ab15-6aecbf49c32f","Type":"ContainerStarted","Data":"2b90e71f9155487ea9de868bfa7cfeac18023f0b8efe1d55653779928d4ce2f9"}
Sep 10 23:18:39.766011 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.765919    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:39.767587 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.767561    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5" event={"ID":"e78b32c4-c119-482c-9f62-dfc4f2ce428f","Type":"ContainerStarted","Data":"e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e"}
Sep 10 23:18:39.767726 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.767604    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5" event={"ID":"e78b32c4-c119-482c-9f62-dfc4f2ce428f","Type":"ContainerStarted","Data":"e552d68666550510f270d8d9473e09584b3d2bb5e6c38e726f86ec8866d47740"}
Sep 10 23:18:39.767726 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.767638    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:39.782769 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.782740    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:39.783731 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.783692    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87" podStartSLOduration=1.7836729999999998 podStartE2EDuration="1.783673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:39.780407892 +0000 UTC m=+1236.216462666" watchObservedRunningTime="2026-09-10 23:18:39.783673 +0000 UTC m=+1236.219727774"
Sep 10 23:18:39.795720 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.795677    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5" podStartSLOduration=1.795663733 podStartE2EDuration="1.795663733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:39.795209381 +0000 UTC m=+1236.231264159" watchObservedRunningTime="2026-09-10 23:18:39.795663733 +0000 UTC m=+1236.231718548"
Sep 10 23:18:39.882023 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:39.881986    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:40.138685 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.138597    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:40.237438 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.237396    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:40.252164 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.252135    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"]
Sep 10 23:18:40.257442 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.257419    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.259594 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.259567    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-6d5278a1-dockercfg-776kl\""
Sep 10 23:18:40.259760 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.259613    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-6d5278a1\""
Sep 10 23:18:40.263652 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.263632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"]
Sep 10 23:18:40.430844 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.430809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f786f697-40a8-43bc-8228-c332c556f9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.431025 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.430864    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sgr2\" (UniqueName: \"kubernetes.io/projected/f786f697-40a8-43bc-8228-c332c556f9c2-kube-api-access-8sgr2\") pod \"payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.532174 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.532135    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f786f697-40a8-43bc-8228-c332c556f9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.532342 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.532199    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sgr2\" (UniqueName: \"kubernetes.io/projected/f786f697-40a8-43bc-8228-c332c556f9c2-kube-api-access-8sgr2\") pod \"payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.532988 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.532963    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f786f697-40a8-43bc-8228-c332c556f9c2-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.540358 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.540333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sgr2\" (UniqueName: \"kubernetes.io/projected/f786f697-40a8-43bc-8228-c332c556f9c2-kube-api-access-8sgr2\") pod \"payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.572110 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.572082    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:40.774727 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.774638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" event={"ID":"cf228744-5cd6-4bf6-bf78-d67a476f2dd2","Type":"ContainerStarted","Data":"65d02fe2e922ac8c409db4d766e1a4de27996b7fd1a2efd492f041e5e62ad247"}
Sep 10 23:18:40.792282 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.792235    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" podStartSLOduration=3.200181405 podStartE2EDuration="3.792220584s" totalImagesPullingTime="592.039179ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:40.790179577 +0000 UTC m=+1237.226234353" watchObservedRunningTime="2026-09-10 23:18:40.792220584 +0000 UTC m=+1237.228275357"
Sep 10 23:18:40.849730 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:40.849513    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"]
Sep 10 23:18:40.852207 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:18:40.852177    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf786f697_40a8_43bc_8228_c332c556f9c2.slice/crio-1db828d55cc5542aefc7ed73c607296b4583b89bbc4f909ebe479775d267648a WatchSource:0}: Error finding container 1db828d55cc5542aefc7ed73c607296b4583b89bbc4f909ebe479775d267648a: Status 404 returned error can't find the container with id 1db828d55cc5542aefc7ed73c607296b4583b89bbc4f909ebe479775d267648a
Sep 10 23:18:41.780792 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:41.780738    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx" event={"ID":"f786f697-40a8-43bc-8228-c332c556f9c2","Type":"ContainerStarted","Data":"8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79"}
Sep 10 23:18:41.781232 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:41.780810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx" event={"ID":"f786f697-40a8-43bc-8228-c332c556f9c2","Type":"ContainerStarted","Data":"1db828d55cc5542aefc7ed73c607296b4583b89bbc4f909ebe479775d267648a"}
Sep 10 23:18:41.781232 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:41.781025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:41.781232 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:41.781068    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:41.798902 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:41.798855    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx" podStartSLOduration=1.798842397 podStartE2EDuration="1.798842397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:18:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:18:41.796738986 +0000 UTC m=+1238.232793759" watchObservedRunningTime="2026-09-10 23:18:41.798842397 +0000 UTC m=+1238.234897169"
Sep 10 23:18:47.791476 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:47.791443    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:18:50.776710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:50.776678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:18:50.777339 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:50.777255    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:18:52.787722 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:52.787689    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:18:53.018342 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:53.018308    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:18:58.610223 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:18:58.610122    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:01.380459 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:19:01.380421    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:01.481279 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:19:01.481237    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:03.390902 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:19:03.390866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:22.678933 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:19:22.678893    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:27.652953 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:19:27.652915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:27.751926 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:19:27.751896    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:19:33.979331 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:19:33.979291    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:23.584106 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:23.584065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.204261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.204175    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.232358 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.232324    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"]
Sep 10 23:20:34.232638 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.232586    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx" podUID="f786f697-40a8-43bc-8228-c332c556f9c2" containerName="payload-pre-processing" containerID="cri-o://8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79" gracePeriod=30
Sep 10 23:20:34.305109 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.305065    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.407630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.407591    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.480484 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.480457    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:20:34.505117 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.505073    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:34.551306 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.551267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f786f697-40a8-43bc-8228-c332c556f9c2-plugins-config-volume\") pod \"f786f697-40a8-43bc-8228-c332c556f9c2\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") "
Sep 10 23:20:34.551495 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.551322    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8sgr2\" (UniqueName: \"kubernetes.io/projected/f786f697-40a8-43bc-8228-c332c556f9c2-kube-api-access-8sgr2\") pod \"f786f697-40a8-43bc-8228-c332c556f9c2\" (UID: \"f786f697-40a8-43bc-8228-c332c556f9c2\") "
Sep 10 23:20:34.551700 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.551677    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f786f697-40a8-43bc-8228-c332c556f9c2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f786f697-40a8-43bc-8228-c332c556f9c2" (UID: "f786f697-40a8-43bc-8228-c332c556f9c2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:20:34.553444 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.553420    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f786f697-40a8-43bc-8228-c332c556f9c2-kube-api-access-8sgr2" (OuterVolumeSpecName: "kube-api-access-8sgr2") pod "f786f697-40a8-43bc-8228-c332c556f9c2" (UID: "f786f697-40a8-43bc-8228-c332c556f9c2"). InnerVolumeSpecName "kube-api-access-8sgr2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:20:34.652170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.652133    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f786f697-40a8-43bc-8228-c332c556f9c2-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:20:34.652170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:34.652163    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sgr2\" (UniqueName: \"kubernetes.io/projected/f786f697-40a8-43bc-8228-c332c556f9c2-kube-api-access-8sgr2\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:20:35.322617 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.322587    2577 generic.go:356] "Generic (PLEG): container finished" podID="f786f697-40a8-43bc-8228-c332c556f9c2" containerID="8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79" exitCode=0
Sep 10 23:20:35.323068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.322657    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"
Sep 10 23:20:35.323068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.322676    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx" event={"ID":"f786f697-40a8-43bc-8228-c332c556f9c2","Type":"ContainerDied","Data":"8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79"}
Sep 10 23:20:35.323068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.322744    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx" event={"ID":"f786f697-40a8-43bc-8228-c332c556f9c2","Type":"ContainerDied","Data":"1db828d55cc5542aefc7ed73c607296b4583b89bbc4f909ebe479775d267648a"}
Sep 10 23:20:35.323068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.322771    2577 scope.go:117] "RemoveContainer" containerID="8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79"
Sep 10 23:20:35.331496 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.331476    2577 scope.go:117] "RemoveContainer" containerID="8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79"
Sep 10 23:20:35.331809 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:20:35.331762    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79\": container with ID starting with 8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79 not found: ID does not exist" containerID="8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79"
Sep 10 23:20:35.331873 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.331808    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79"} err="failed to get container status \"8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79\": rpc error: code = NotFound desc = could not find container \"8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79\": container with ID starting with 8ba7b2873a7bc18dd482d20c9b40dde94cb019ca143c55456616e08d9e5d2a79 not found: ID does not exist"
Sep 10 23:20:35.344121 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.344098    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"]
Sep 10 23:20:35.348100 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:35.348078    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-6d5278a1-75bd944fdb-ktlnx"]
Sep 10 23:20:36.215820 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:36.215763    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f786f697-40a8-43bc-8228-c332c556f9c2" path="/var/lib/kubelet/pods/f786f697-40a8-43bc-8228-c332c556f9c2/volumes"
Sep 10 23:20:37.548387 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:37.548346    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:42.569271 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:42.569220    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:20:53.627691 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:20:53.627658    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:18.489844 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:18.489808    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:18.591189 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:18.591142    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:19.665038 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:19.665004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:19.769517 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:19.769477    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:25.597611 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:25.597569    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:31.413427 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:31.413384    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:31.467319 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:31.467287    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:31.685177 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:31.685084    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:32.913966 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:32.913924    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.013011 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:33.012981    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.113886 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:33.113845    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.673027 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:33.672980    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.769963 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:33.769933    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:33.869904 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:33.869866    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:35.101249 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:35.101214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:41.185386 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:41.185350    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:41.285846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:41.285813    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:41.836295 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:41.836257    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:48.071354 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:48.071311    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:48.171067 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:48.171023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:48.271068 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:48.271009    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:58.734082 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:58.733996    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:21:58.957212 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:21:58.957173    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:16.416305 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:16.416266    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:17.497437 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:17.497398    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:19.872176 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:19.872131    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:22.983557 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:22.983522    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:26.261965 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:26.261915    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:26.325877 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:26.325837    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:26.596348 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:26.596250    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:26.899900 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:26.899801    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:27.607886 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:27.607844    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:29.936911 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:29.936873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:30.034284 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:30.034242    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.182136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.182094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.279979 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.279947    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.379940 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.379902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.631767 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.631663    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.727081 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.727042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:31.728117 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.728091    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"]
Sep 10 23:22:31.728723 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.728703    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f786f697-40a8-43bc-8228-c332c556f9c2" containerName="payload-pre-processing"
Sep 10 23:22:31.728723 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.728723    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f786f697-40a8-43bc-8228-c332c556f9c2" containerName="payload-pre-processing"
Sep 10 23:22:31.728897 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.728827    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f786f697-40a8-43bc-8228-c332c556f9c2" containerName="payload-pre-processing"
Sep 10 23:22:31.732066 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.732048    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:31.734010 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.733990    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5052bd\""
Sep 10 23:22:31.741324 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.741284    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"]
Sep 10 23:22:31.839016 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.838981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:31.839208 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.839101    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkc6r\" (UniqueName: \"kubernetes.io/projected/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-kube-api-access-nkc6r\") pod \"payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:31.940381 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.940335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkc6r\" (UniqueName: \"kubernetes.io/projected/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-kube-api-access-nkc6r\") pod \"payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:31.940602 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.940427    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:31.941193 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.941169    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:31.951376 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:31.951348    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkc6r\" (UniqueName: \"kubernetes.io/projected/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-kube-api-access-nkc6r\") pod \"payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:32.044089 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:32.044054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:32.322453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:32.322376    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"]
Sep 10 23:22:32.325099 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:22:32.325067    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d83255b_0dd3_4462_a1d9_ad5185f6afa7.slice/crio-ad698cf4bc54e114e14e550b01d9151135f6e07c3ac09c47312dad0659e0bc2d WatchSource:0}: Error finding container ad698cf4bc54e114e14e550b01d9151135f6e07c3ac09c47312dad0659e0bc2d: Status 404 returned error can't find the container with id ad698cf4bc54e114e14e550b01d9151135f6e07c3ac09c47312dad0659e0bc2d
Sep 10 23:22:32.924531 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:32.924494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp" event={"ID":"3d83255b-0dd3-4462-a1d9-ad5185f6afa7","Type":"ContainerStarted","Data":"aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b"}
Sep 10 23:22:32.924727 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:32.924552    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp" event={"ID":"3d83255b-0dd3-4462-a1d9-ad5185f6afa7","Type":"ContainerStarted","Data":"ad698cf4bc54e114e14e550b01d9151135f6e07c3ac09c47312dad0659e0bc2d"}
Sep 10 23:22:32.924727 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:32.924589    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:32.941576 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:32.941519    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp" podStartSLOduration=1.9415023059999998 podStartE2EDuration="1.941502306s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:22:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:22:32.939982976 +0000 UTC m=+1469.376037748" watchObservedRunningTime="2026-09-10 23:22:32.941502306 +0000 UTC m=+1469.377557078"
Sep 10 23:22:43.932268 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:43.932233    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:22:53.098967 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:53.098930    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:53.645358 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:53.645314    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:54.918979 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:54.918948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:55.393211 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:55.393094    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:55.700347 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:55.700298    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:22:56.052111 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:22:56.052004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:01.566532 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:01.566490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:01.669283 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:01.669235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:02.806853 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:02.806775    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:02.867986 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:02.867946    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:02.969835 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:02.969769    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:03.399429 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.399390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:03.505555 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.505520    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:03.506687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.506664    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"]
Sep 10 23:23:03.510276 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.510259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:03.512524 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.512500    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-44c67e-dockercfg-94j9b\""
Sep 10 23:23:03.512636 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.512528    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-44c67e\""
Sep 10 23:23:03.518193 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.518161    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"]
Sep 10 23:23:03.635476 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.635435    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6d9d68-758b-4578-99b4-836157c6cfd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:03.635826 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.635768    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgkxn\" (UniqueName: \"kubernetes.io/projected/5d6d9d68-758b-4578-99b4-836157c6cfd3-kube-api-access-dgkxn\") pod \"payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:03.737507 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.737463    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgkxn\" (UniqueName: \"kubernetes.io/projected/5d6d9d68-758b-4578-99b4-836157c6cfd3-kube-api-access-dgkxn\") pod \"payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:03.737734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.737601    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6d9d68-758b-4578-99b4-836157c6cfd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:03.738532 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.738506    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6d9d68-758b-4578-99b4-836157c6cfd3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:03.745566 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.745535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgkxn\" (UniqueName: \"kubernetes.io/projected/5d6d9d68-758b-4578-99b4-836157c6cfd3-kube-api-access-dgkxn\") pod \"payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:03.822000 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:03.821953    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:04.100604 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:04.100570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"]
Sep 10 23:23:04.102429 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:23:04.102402    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d6d9d68_758b_4578_99b4_836157c6cfd3.slice/crio-452502b17b4cb917da67f4f81c87e3f880dbd0af54c55f24e1978455fb666f10 WatchSource:0}: Error finding container 452502b17b4cb917da67f4f81c87e3f880dbd0af54c55f24e1978455fb666f10: Status 404 returned error can't find the container with id 452502b17b4cb917da67f4f81c87e3f880dbd0af54c55f24e1978455fb666f10
Sep 10 23:23:05.082361 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:05.082320    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6" event={"ID":"5d6d9d68-758b-4578-99b4-836157c6cfd3","Type":"ContainerStarted","Data":"8b39f13f22338b283781957f1a3f5d2e2349520f332f4529f576a3829fec9f95"}
Sep 10 23:23:05.082751 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:05.082377    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6" event={"ID":"5d6d9d68-758b-4578-99b4-836157c6cfd3","Type":"ContainerStarted","Data":"452502b17b4cb917da67f4f81c87e3f880dbd0af54c55f24e1978455fb666f10"}
Sep 10 23:23:05.082751 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:05.082415    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:05.099831 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:05.099765    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6" podStartSLOduration=2.099749343 podStartE2EDuration="2.099749343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:23:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:23:05.097899374 +0000 UTC m=+1501.533954172" watchObservedRunningTime="2026-09-10 23:23:05.099749343 +0000 UTC m=+1501.535804149"
Sep 10 23:23:16.089378 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:16.089342    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:23:23.102868 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:23.102810    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:23.831187 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:23.831151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:26.543657 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:26.543573    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:30.094458 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:30.094423    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:31.136943 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:31.136904    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:33.139044 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:33.139012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:33.435948 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:33.435910    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:34.448277 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:34.448238    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:36.671953 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:36.671886    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:39.606206 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:39.606171    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:45.172764 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.172723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:45.215316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.215275    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"]
Sep 10 23:23:45.215519 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.215488    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp" podUID="3d83255b-0dd3-4462-a1d9-ad5185f6afa7" containerName="payload-pre-processing" containerID="cri-o://aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b" gracePeriod=30
Sep 10 23:23:45.243422 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.243390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:45.305908 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.305871    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:45.367644 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.367598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:45.426246 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.426161    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:45.487052 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.487012    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:23:45.628923 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.628890    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-plugins-config-volume\") pod \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") "
Sep 10 23:23:45.629123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.628995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nkc6r\" (UniqueName: \"kubernetes.io/projected/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-kube-api-access-nkc6r\") pod \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\" (UID: \"3d83255b-0dd3-4462-a1d9-ad5185f6afa7\") "
Sep 10 23:23:45.629351 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.629323    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3d83255b-0dd3-4462-a1d9-ad5185f6afa7" (UID: "3d83255b-0dd3-4462-a1d9-ad5185f6afa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:23:45.631116 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.631095    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-kube-api-access-nkc6r" (OuterVolumeSpecName: "kube-api-access-nkc6r") pod "3d83255b-0dd3-4462-a1d9-ad5185f6afa7" (UID: "3d83255b-0dd3-4462-a1d9-ad5185f6afa7"). InnerVolumeSpecName "kube-api-access-nkc6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:23:45.704860 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.704765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:45.729958 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.729931    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:23:45.729958 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:45.729955    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkc6r\" (UniqueName: \"kubernetes.io/projected/3d83255b-0dd3-4462-a1d9-ad5185f6afa7-kube-api-access-nkc6r\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:23:46.284453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.284419    2577 generic.go:356] "Generic (PLEG): container finished" podID="3d83255b-0dd3-4462-a1d9-ad5185f6afa7" containerID="aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b" exitCode=0
Sep 10 23:23:46.284996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.284467    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"
Sep 10 23:23:46.284996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.284504    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp" event={"ID":"3d83255b-0dd3-4462-a1d9-ad5185f6afa7","Type":"ContainerDied","Data":"aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b"}
Sep 10 23:23:46.284996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.284566    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp" event={"ID":"3d83255b-0dd3-4462-a1d9-ad5185f6afa7","Type":"ContainerDied","Data":"ad698cf4bc54e114e14e550b01d9151135f6e07c3ac09c47312dad0659e0bc2d"}
Sep 10 23:23:46.284996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.284609    2577 scope.go:117] "RemoveContainer" containerID="aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b"
Sep 10 23:23:46.293357 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.293340    2577 scope.go:117] "RemoveContainer" containerID="aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b"
Sep 10 23:23:46.293634 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:23:46.293616    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b\": container with ID starting with aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b not found: ID does not exist" containerID="aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b"
Sep 10 23:23:46.293689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.293643    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b"} err="failed to get container status \"aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b\": rpc error: code = NotFound desc = could not find container \"aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b\": container with ID starting with aefeeed8c176114e95d552dc395c8fbb2cdfe4a79bfade21c10ca31b0144762b not found: ID does not exist"
Sep 10 23:23:46.301391 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.301368    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"]
Sep 10 23:23:46.305085 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:46.305064    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5052bd-5fb697c6fd-hn5jp"]
Sep 10 23:23:47.529166 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:47.529130    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:48.214302 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:48.214269    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d83255b-0dd3-4462-a1d9-ad5185f6afa7" path="/var/lib/kubelet/pods/3d83255b-0dd3-4462-a1d9-ad5185f6afa7/volumes"
Sep 10 23:23:51.638150 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:51.638118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:51.703617 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:51.703581    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:51.886825 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:51.886764    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:52.307245 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:52.307201    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:53.911473 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:53.911408    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:23:59.249173 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:23:59.249132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:02.492757 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:02.492717    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:07.190000 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:07.189963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:08.165594 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.165561    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:08.198161 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.198128    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"]
Sep 10 23:24:08.198610 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.198392    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6" podUID="5d6d9d68-758b-4578-99b4-836157c6cfd3" containerName="payload-pre-processing" containerID="cri-o://8b39f13f22338b283781957f1a3f5d2e2349520f332f4529f576a3829fec9f95" gracePeriod=30
Sep 10 23:24:08.222071 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.222039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:08.276723 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.276683    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:08.356381 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.356345    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:08.395367 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.395319    2577 generic.go:356] "Generic (PLEG): container finished" podID="5d6d9d68-758b-4578-99b4-836157c6cfd3" containerID="8b39f13f22338b283781957f1a3f5d2e2349520f332f4529f576a3829fec9f95" exitCode=0
Sep 10 23:24:08.395558 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.395494    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6" event={"ID":"5d6d9d68-758b-4578-99b4-836157c6cfd3","Type":"ContainerDied","Data":"8b39f13f22338b283781957f1a3f5d2e2349520f332f4529f576a3829fec9f95"}
Sep 10 23:24:08.417757 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.417662    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:08.461876 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.461843    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:24:08.542206 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.542174    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgkxn\" (UniqueName: \"kubernetes.io/projected/5d6d9d68-758b-4578-99b4-836157c6cfd3-kube-api-access-dgkxn\") pod \"5d6d9d68-758b-4578-99b4-836157c6cfd3\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") "
Sep 10 23:24:08.542479 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.542251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6d9d68-758b-4578-99b4-836157c6cfd3-plugins-config-volume\") pod \"5d6d9d68-758b-4578-99b4-836157c6cfd3\" (UID: \"5d6d9d68-758b-4578-99b4-836157c6cfd3\") "
Sep 10 23:24:08.542700 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.542676    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d6d9d68-758b-4578-99b4-836157c6cfd3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5d6d9d68-758b-4578-99b4-836157c6cfd3" (UID: "5d6d9d68-758b-4578-99b4-836157c6cfd3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:24:08.544447 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.544423    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d6d9d68-758b-4578-99b4-836157c6cfd3-kube-api-access-dgkxn" (OuterVolumeSpecName: "kube-api-access-dgkxn") pod "5d6d9d68-758b-4578-99b4-836157c6cfd3" (UID: "5d6d9d68-758b-4578-99b4-836157c6cfd3"). InnerVolumeSpecName "kube-api-access-dgkxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:24:08.643981 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.643945    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgkxn\" (UniqueName: \"kubernetes.io/projected/5d6d9d68-758b-4578-99b4-836157c6cfd3-kube-api-access-dgkxn\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:08.643981 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.643974    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5d6d9d68-758b-4578-99b4-836157c6cfd3-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:08.698123 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:08.698036    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:09.401823 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:09.401774    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6" event={"ID":"5d6d9d68-758b-4578-99b4-836157c6cfd3","Type":"ContainerDied","Data":"452502b17b4cb917da67f4f81c87e3f880dbd0af54c55f24e1978455fb666f10"}
Sep 10 23:24:09.401823 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:09.401821    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"
Sep 10 23:24:09.402317 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:09.401853    2577 scope.go:117] "RemoveContainer" containerID="8b39f13f22338b283781957f1a3f5d2e2349520f332f4529f576a3829fec9f95"
Sep 10 23:24:09.428274 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:09.428247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"]
Sep 10 23:24:09.432401 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:09.432378    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-44c67e-9c9d7fbdb-kjwf6"]
Sep 10 23:24:10.214133 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:10.214098    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d6d9d68-758b-4578-99b4-836157c6cfd3" path="/var/lib/kubelet/pods/5d6d9d68-758b-4578-99b4-836157c6cfd3/volumes"
Sep 10 23:24:14.586728 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:14.586685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:14.686700 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:14.686664    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:14.858557 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:14.858466    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:15.303255 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:15.303218    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:25.592000 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.591966    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"]
Sep 10 23:24:25.592467 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.592248    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p" podUID="a6b8f98b-9f4b-41c8-91c2-845d267b8259" containerName="maas-api" containerID="cri-o://f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7" gracePeriod=30
Sep 10 23:24:25.617590 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.617563    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:25.678281 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.678244    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:25.729924 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.729888    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:25.804519 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.804490    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:25.835541 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.835517    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:24:25.863282 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:25.863214    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:26.004529 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.004495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-maas-api-tls\") pod \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") "
Sep 10 23:24:26.004697 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.004588    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8f98b-9f4b-41c8-91c2-845d267b8259-openshift-service-ca\") pod \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") "
Sep 10 23:24:26.004697 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.004633    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m8lkm\" (UniqueName: \"kubernetes.io/projected/a6b8f98b-9f4b-41c8-91c2-845d267b8259-kube-api-access-m8lkm\") pod \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") "
Sep 10 23:24:26.004697 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.004693    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-metrics-certs\") pod \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\" (UID: \"a6b8f98b-9f4b-41c8-91c2-845d267b8259\") "
Sep 10 23:24:26.005025 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.004990    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6b8f98b-9f4b-41c8-91c2-845d267b8259-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a6b8f98b-9f4b-41c8-91c2-845d267b8259" (UID: "a6b8f98b-9f4b-41c8-91c2-845d267b8259"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:24:26.006901 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.006870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a6b8f98b-9f4b-41c8-91c2-845d267b8259" (UID: "a6b8f98b-9f4b-41c8-91c2-845d267b8259"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:24:26.006977 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.006928    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a6b8f98b-9f4b-41c8-91c2-845d267b8259" (UID: "a6b8f98b-9f4b-41c8-91c2-845d267b8259"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:24:26.007023 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.006972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6b8f98b-9f4b-41c8-91c2-845d267b8259-kube-api-access-m8lkm" (OuterVolumeSpecName: "kube-api-access-m8lkm") pod "a6b8f98b-9f4b-41c8-91c2-845d267b8259" (UID: "a6b8f98b-9f4b-41c8-91c2-845d267b8259"). InnerVolumeSpecName "kube-api-access-m8lkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:24:26.105819 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.105772    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6b8f98b-9f4b-41c8-91c2-845d267b8259-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:26.105819 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.105815    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8lkm\" (UniqueName: \"kubernetes.io/projected/a6b8f98b-9f4b-41c8-91c2-845d267b8259-kube-api-access-m8lkm\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:26.105819 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.105825    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:26.106007 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.105833    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6b8f98b-9f4b-41c8-91c2-845d267b8259-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:26.174436 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.174401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:26.485971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.485876    2577 generic.go:356] "Generic (PLEG): container finished" podID="a6b8f98b-9f4b-41c8-91c2-845d267b8259" containerID="f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7" exitCode=0
Sep 10 23:24:26.485971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.485926    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"
Sep 10 23:24:26.485971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.485958    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p" event={"ID":"a6b8f98b-9f4b-41c8-91c2-845d267b8259","Type":"ContainerDied","Data":"f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7"}
Sep 10 23:24:26.486193 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.486018    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p" event={"ID":"a6b8f98b-9f4b-41c8-91c2-845d267b8259","Type":"ContainerDied","Data":"5ada2e7233d7833a69baf45b7cf447b46722c67980f8decfba6b621262350077"}
Sep 10 23:24:26.486193 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.486041    2577 scope.go:117] "RemoveContainer" containerID="f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7"
Sep 10 23:24:26.495523 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.495503    2577 scope.go:117] "RemoveContainer" containerID="f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7"
Sep 10 23:24:26.495833 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:24:26.495813    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7\": container with ID starting with f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7 not found: ID does not exist" containerID="f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7"
Sep 10 23:24:26.495888 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.495843    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7"} err="failed to get container status \"f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7\": rpc error: code = NotFound desc = could not find container \"f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7\": container with ID starting with f880a1d3e2f035226efc4fd353f44aadb0bddf586313e54fb9692502d7019bc7 not found: ID does not exist"
Sep 10 23:24:26.503988 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.503962    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"]
Sep 10 23:24:26.508240 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:26.508215    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-94f305-ffcdd868-ltm4p"]
Sep 10 23:24:28.214681 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:28.214647    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6b8f98b-9f4b-41c8-91c2-845d267b8259" path="/var/lib/kubelet/pods/a6b8f98b-9f4b-41c8-91c2-845d267b8259/volumes"
Sep 10 23:24:32.255667 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:32.255627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:32.355792 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:32.355754    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:32.513062 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:32.512952    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:33.105505 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:33.105469    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:43.405566 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.405525    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"]
Sep 10 23:24:43.406116 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.405842    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4" podUID="1af934bd-1d1b-4b3d-beeb-50322b80155d" containerName="maas-api" containerID="cri-o://b806be658d8d778b2469fca91794d08f27d7ae1dce72713bc9d0cd59611f0748" gracePeriod=30
Sep 10 23:24:43.429315 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.429280    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:43.458815 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.458761    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"]
Sep 10 23:24:43.459021 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.458983    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q" podUID="6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" containerName="payload-pre-processing" containerID="cri-o://df9b1b92cf5521f820181f7208ae0335797dd3d89e2b0e10e1a3a358d6e44743" gracePeriod=30
Sep 10 23:24:43.490410 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.490377    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:43.574196 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.574170    2577 generic.go:356] "Generic (PLEG): container finished" podID="1af934bd-1d1b-4b3d-beeb-50322b80155d" containerID="b806be658d8d778b2469fca91794d08f27d7ae1dce72713bc9d0cd59611f0748" exitCode=0
Sep 10 23:24:43.574299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.574245    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4" event={"ID":"1af934bd-1d1b-4b3d-beeb-50322b80155d","Type":"ContainerDied","Data":"b806be658d8d778b2469fca91794d08f27d7ae1dce72713bc9d0cd59611f0748"}
Sep 10 23:24:43.576058 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.576031    2577 generic.go:356] "Generic (PLEG): container finished" podID="6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" containerID="df9b1b92cf5521f820181f7208ae0335797dd3d89e2b0e10e1a3a358d6e44743" exitCode=0
Sep 10 23:24:43.576169 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.576101    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q" event={"ID":"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7","Type":"ContainerDied","Data":"df9b1b92cf5521f820181f7208ae0335797dd3d89e2b0e10e1a3a358d6e44743"}
Sep 10 23:24:43.590673 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.590641    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:43.647628 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.647565    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:43.717795 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.717758    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:24:43.739051 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.739025    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:24:43.760818 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.760771    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-plugins-config-volume\") pod \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") "
Sep 10 23:24:43.760989 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.760854    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hl5c5\" (UniqueName: \"kubernetes.io/projected/1af934bd-1d1b-4b3d-beeb-50322b80155d-kube-api-access-hl5c5\") pod \"1af934bd-1d1b-4b3d-beeb-50322b80155d\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") "
Sep 10 23:24:43.761184 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.761158    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" (UID: "6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:24:43.761278 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.761211    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ln2bl\" (UniqueName: \"kubernetes.io/projected/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-kube-api-access-ln2bl\") pod \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\" (UID: \"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7\") "
Sep 10 23:24:43.761278 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.761267    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-metrics-certs\") pod \"1af934bd-1d1b-4b3d-beeb-50322b80155d\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") "
Sep 10 23:24:43.761387 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.761298    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af934bd-1d1b-4b3d-beeb-50322b80155d-openshift-service-ca\") pod \"1af934bd-1d1b-4b3d-beeb-50322b80155d\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") "
Sep 10 23:24:43.761387 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.761355    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-maas-api-tls\") pod \"1af934bd-1d1b-4b3d-beeb-50322b80155d\" (UID: \"1af934bd-1d1b-4b3d-beeb-50322b80155d\") "
Sep 10 23:24:43.761734 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.761712    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:43.762013 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.761987    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1af934bd-1d1b-4b3d-beeb-50322b80155d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1af934bd-1d1b-4b3d-beeb-50322b80155d" (UID: "1af934bd-1d1b-4b3d-beeb-50322b80155d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:24:43.763292 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.763266    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1af934bd-1d1b-4b3d-beeb-50322b80155d-kube-api-access-hl5c5" (OuterVolumeSpecName: "kube-api-access-hl5c5") pod "1af934bd-1d1b-4b3d-beeb-50322b80155d" (UID: "1af934bd-1d1b-4b3d-beeb-50322b80155d"). InnerVolumeSpecName "kube-api-access-hl5c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:24:43.763390 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.763370    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-kube-api-access-ln2bl" (OuterVolumeSpecName: "kube-api-access-ln2bl") pod "6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" (UID: "6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7"). InnerVolumeSpecName "kube-api-access-ln2bl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:24:43.763568 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.763549    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1af934bd-1d1b-4b3d-beeb-50322b80155d" (UID: "1af934bd-1d1b-4b3d-beeb-50322b80155d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:24:43.763951 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.763924    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1af934bd-1d1b-4b3d-beeb-50322b80155d" (UID: "1af934bd-1d1b-4b3d-beeb-50322b80155d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:24:43.862983 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.862937    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hl5c5\" (UniqueName: \"kubernetes.io/projected/1af934bd-1d1b-4b3d-beeb-50322b80155d-kube-api-access-hl5c5\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:43.862983 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.862974    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ln2bl\" (UniqueName: \"kubernetes.io/projected/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7-kube-api-access-ln2bl\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:43.862983 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.862987    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:43.863222 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.862998    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1af934bd-1d1b-4b3d-beeb-50322b80155d-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:43.863222 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.863011    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1af934bd-1d1b-4b3d-beeb-50322b80155d-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:24:43.963344 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:43.963246    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:44.581558 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.581470    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"
Sep 10 23:24:44.581558 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.581492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4" event={"ID":"1af934bd-1d1b-4b3d-beeb-50322b80155d","Type":"ContainerDied","Data":"e6c6c7dd833c8a208f9cc09c98be7ffff6741b3c25092d18922f93dabac46fb4"}
Sep 10 23:24:44.581558 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.581555    2577 scope.go:117] "RemoveContainer" containerID="b806be658d8d778b2469fca91794d08f27d7ae1dce72713bc9d0cd59611f0748"
Sep 10 23:24:44.582969 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.582946    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q" event={"ID":"6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7","Type":"ContainerDied","Data":"7b2f352bb9e408983fd68c0a27fdde4473b5fde6c283f09b1bfb9ab1536bcf05"}
Sep 10 23:24:44.583042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.582956    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"
Sep 10 23:24:44.590506 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.590476    2577 scope.go:117] "RemoveContainer" containerID="df9b1b92cf5521f820181f7208ae0335797dd3d89e2b0e10e1a3a358d6e44743"
Sep 10 23:24:44.598486 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.598466    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"]
Sep 10 23:24:44.602596 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.602575    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-063e1d-68b47b9c9c-k22q4"]
Sep 10 23:24:44.613201 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.613176    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"]
Sep 10 23:24:44.617116 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:44.617093    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-063e1d-68b55df946-wvw7q"]
Sep 10 23:24:46.214424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:46.214390    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1af934bd-1d1b-4b3d-beeb-50322b80155d" path="/var/lib/kubelet/pods/1af934bd-1d1b-4b3d-beeb-50322b80155d/volumes"
Sep 10 23:24:46.214812 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:46.214747    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" path="/var/lib/kubelet/pods/6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7/volumes"
Sep 10 23:24:50.323731 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:50.323685    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:50.386818 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:50.386766    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:24:50.574089 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:24:50.573994    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:00.159063 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:00.158984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:01.168230 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:01.168198    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:01.596503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:01.596400    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:01.896479 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:01.896386    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:02.324501 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:02.324464    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:05.001273 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:05.001230    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:05.093554 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:05.093518    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:06.312772 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:06.312734    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:06.412195 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:06.412156    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:06.465419 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:06.465374    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:06.791737 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:06.791696    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:06.882618 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:06.882577    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:15.650386 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:15.650348    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:29.958153 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:29.958114    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:30.615192 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:30.615151    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:30.985056 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:30.985012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:31.084909 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:31.084868    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:34.438454 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:34.438417    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:34.540632 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:34.540584    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:35.586504 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.586469    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"]
Sep 10 23:25:35.587070 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587052    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d6d9d68-758b-4578-99b4-836157c6cfd3" containerName="payload-pre-processing"
Sep 10 23:25:35.587134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587072    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d6d9d68-758b-4578-99b4-836157c6cfd3" containerName="payload-pre-processing"
Sep 10 23:25:35.587134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587087    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1af934bd-1d1b-4b3d-beeb-50322b80155d" containerName="maas-api"
Sep 10 23:25:35.587134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587096    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1af934bd-1d1b-4b3d-beeb-50322b80155d" containerName="maas-api"
Sep 10 23:25:35.587134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587107    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d83255b-0dd3-4462-a1d9-ad5185f6afa7" containerName="payload-pre-processing"
Sep 10 23:25:35.587134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587115    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d83255b-0dd3-4462-a1d9-ad5185f6afa7" containerName="payload-pre-processing"
Sep 10 23:25:35.587134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587129    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" containerName="payload-pre-processing"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587137    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" containerName="payload-pre-processing"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6b8f98b-9f4b-41c8-91c2-845d267b8259" containerName="maas-api"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587168    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6b8f98b-9f4b-41c8-91c2-845d267b8259" containerName="maas-api"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587245    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1af934bd-1d1b-4b3d-beeb-50322b80155d" containerName="maas-api"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587259    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d6d9d68-758b-4578-99b4-836157c6cfd3" containerName="payload-pre-processing"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587268    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dcb74d8-d774-4c31-bd2b-2c6a9e197aa7" containerName="payload-pre-processing"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587276    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6b8f98b-9f4b-41c8-91c2-845d267b8259" containerName="maas-api"
Sep 10 23:25:35.587316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.587285    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d83255b-0dd3-4462-a1d9-ad5185f6afa7" containerName="payload-pre-processing"
Sep 10 23:25:35.590581 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.590560    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.593322 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.593292    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-c6c079-serving-cert\""
Sep 10 23:25:35.603453 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.603432    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"]
Sep 10 23:25:35.611657 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.611627    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:35.624226 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.624201    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.624345 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.624302    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7slfw\" (UniqueName: \"kubernetes.io/projected/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-kube-api-access-7slfw\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.624408 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.624352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-metrics-certs\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.624460 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.624417    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.708423 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.708383    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:35.725432 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.725401    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.725577 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.725460    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.725577 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.725519    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7slfw\" (UniqueName: \"kubernetes.io/projected/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-kube-api-access-7slfw\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.725577 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.725549    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-metrics-certs\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.725577 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:25:35.725560    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-serving-cert: secret "maas-api-e2e-ipp-b-c6c079-serving-cert" not found
Sep 10 23:25:35.725734 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:25:35.725640    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls podName:9b382f05-4dc8-4c24-9687-d0ccd6064a5a nodeName:}" failed. No retries permitted until 2026-09-10 23:25:36.225617562 +0000 UTC m=+1652.661672312 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls") pod "maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf" (UID: "9b382f05-4dc8-4c24-9687-d0ccd6064a5a") : secret "maas-api-e2e-ipp-b-c6c079-serving-cert" not found
Sep 10 23:25:35.726115 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.726096    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.727990 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.727971    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-metrics-certs\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.739255 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.739229    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7slfw\" (UniqueName: \"kubernetes.io/projected/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-kube-api-access-7slfw\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:35.809644 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:35.809598    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:36.098171 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.098127    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:36.196048 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.196013    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:36.229912 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.229874    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:36.232217 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.232191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls\") pod \"maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:36.500653 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.500614    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:36.771037 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.771012    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"]
Sep 10 23:25:36.773475 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:25:36.773447    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b382f05_4dc8_4c24_9687_d0ccd6064a5a.slice/crio-1acc3afe45ecb90bd6ac54dc8f3ce43c7b6e5b2cc08d7540902b52cd4e4ba33d WatchSource:0}: Error finding container 1acc3afe45ecb90bd6ac54dc8f3ce43c7b6e5b2cc08d7540902b52cd4e4ba33d: Status 404 returned error can't find the container with id 1acc3afe45ecb90bd6ac54dc8f3ce43c7b6e5b2cc08d7540902b52cd4e4ba33d
Sep 10 23:25:36.774843 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.774822    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:25:36.826229 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:36.826202    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf" event={"ID":"9b382f05-4dc8-4c24-9687-d0ccd6064a5a","Type":"ContainerStarted","Data":"1acc3afe45ecb90bd6ac54dc8f3ce43c7b6e5b2cc08d7540902b52cd4e4ba33d"}
Sep 10 23:25:37.832344 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:37.832310    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf" event={"ID":"9b382f05-4dc8-4c24-9687-d0ccd6064a5a","Type":"ContainerStarted","Data":"ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a"}
Sep 10 23:25:37.832752 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:37.832443    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:37.849999 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:37.849955    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf" podStartSLOduration=2.407225547 podStartE2EDuration="2.84994181s" totalImagesPullingTime="442.716263ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:25:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:25:37.848919563 +0000 UTC m=+1654.284974328" watchObservedRunningTime="2026-09-10 23:25:37.84994181 +0000 UTC m=+1654.285996627"
Sep 10 23:25:43.843013 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:43.842984    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:25:46.911573 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:46.911540    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:46.968067 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:46.968042    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:56.235846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:56.235806    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:56.337139 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:56.337104    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:25:59.419049 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:25:59.419006    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:26:16.154279 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:26:16.154234    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:01.171995 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:01.171963    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:01.478437 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:01.478342    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:01.639160 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:01.639118    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:04.952048 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:04.952010    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:05.040939 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:05.040891    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:06.074442 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.074407    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"]
Sep 10 23:27:06.078118 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.078091    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.080106 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.080080    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-2cdd19-serving-cert\""
Sep 10 23:27:06.089273 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.089244    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"]
Sep 10 23:27:06.109971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.109929    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:06.154144 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.154106    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.154331 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.154246    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.154331 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.154311    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lw55\" (UniqueName: \"kubernetes.io/projected/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-kube-api-access-2lw55\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.154407 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.154355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.209985 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.209948    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:06.255045 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.254994    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.255263 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.255220    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.255263 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.255259    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lw55\" (UniqueName: \"kubernetes.io/projected/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-kube-api-access-2lw55\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.255392 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.255303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.255853 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.255824    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.257808 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.257770    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.257966 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.257946    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.263510 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.263483    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lw55\" (UniqueName: \"kubernetes.io/projected/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-kube-api-access-2lw55\") pod \"maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.310191 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.310159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:06.392149 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.392058    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:06.624962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.624920    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:06.669972 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.669945    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"]
Sep 10 23:27:06.672253 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:27:06.672225    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffe5261_3aa5_4532_9a30_dee7e2d69fa3.slice/crio-baf8a4916708c84bc6844dea6c50b0be9ca9fbdb3641145207b5bb5c34264e65 WatchSource:0}: Error finding container baf8a4916708c84bc6844dea6c50b0be9ca9fbdb3641145207b5bb5c34264e65: Status 404 returned error can't find the container with id baf8a4916708c84bc6844dea6c50b0be9ca9fbdb3641145207b5bb5c34264e65
Sep 10 23:27:06.722253 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.722215    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:06.735875 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.735841    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"]
Sep 10 23:27:06.740731 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.740702    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:06.742921 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.742897    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2cdd19-dockercfg-sxzfb\""
Sep 10 23:27:06.743048 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.742969    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2cdd19\""
Sep 10 23:27:06.749321 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.749294    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"]
Sep 10 23:27:06.861583 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.861545    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwjpk\" (UniqueName: \"kubernetes.io/projected/7644980b-fc7f-4c80-9a27-24132523a407-kube-api-access-nwjpk\") pod \"payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:06.861744 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.861638    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7644980b-fc7f-4c80-9a27-24132523a407-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:06.962227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.962134    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7644980b-fc7f-4c80-9a27-24132523a407-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:06.962227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.962195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwjpk\" (UniqueName: \"kubernetes.io/projected/7644980b-fc7f-4c80-9a27-24132523a407-kube-api-access-nwjpk\") pod \"payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:06.962881 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.962862    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7644980b-fc7f-4c80-9a27-24132523a407-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:06.970421 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:06.970399    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwjpk\" (UniqueName: \"kubernetes.io/projected/7644980b-fc7f-4c80-9a27-24132523a407-kube-api-access-nwjpk\") pod \"payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:07.053733 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:07.053707    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:07.248975 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:07.248884    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv" event={"ID":"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3","Type":"ContainerStarted","Data":"5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373"}
Sep 10 23:27:07.248975 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:07.248936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv" event={"ID":"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3","Type":"ContainerStarted","Data":"baf8a4916708c84bc6844dea6c50b0be9ca9fbdb3641145207b5bb5c34264e65"}
Sep 10 23:27:07.249437 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:07.249018    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:07.266736 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:07.266675    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv" podStartSLOduration=0.894497857 podStartE2EDuration="1.26665119s" totalImagesPullingTime="372.153333ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:27:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:27:07.265326069 +0000 UTC m=+1743.701380843" watchObservedRunningTime="2026-09-10 23:27:07.26665119 +0000 UTC m=+1743.702705962"
Sep 10 23:27:07.333494 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:07.333464    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"]
Sep 10 23:27:07.336076 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:27:07.336036    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7644980b_fc7f_4c80_9a27_24132523a407.slice/crio-01d822a66336615374278151180f4266092ed8e7da6f19b7152e14da8ccc5a4e WatchSource:0}: Error finding container 01d822a66336615374278151180f4266092ed8e7da6f19b7152e14da8ccc5a4e: Status 404 returned error can't find the container with id 01d822a66336615374278151180f4266092ed8e7da6f19b7152e14da8ccc5a4e
Sep 10 23:27:08.263849 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:08.263805    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8" event={"ID":"7644980b-fc7f-4c80-9a27-24132523a407","Type":"ContainerStarted","Data":"b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279"}
Sep 10 23:27:08.264238 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:08.264090    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:08.264238 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:08.264134    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8" event={"ID":"7644980b-fc7f-4c80-9a27-24132523a407","Type":"ContainerStarted","Data":"01d822a66336615374278151180f4266092ed8e7da6f19b7152e14da8ccc5a4e"}
Sep 10 23:27:08.280475 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:08.280424    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8" podStartSLOduration=2.280408947 podStartE2EDuration="2.280408947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:27:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:27:08.279362361 +0000 UTC m=+1744.715417145" watchObservedRunningTime="2026-09-10 23:27:08.280408947 +0000 UTC m=+1744.716463773"
Sep 10 23:27:13.269270 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:13.269241    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:19.271307 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:19.271270    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:29.896736 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:29.896702    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:36.790215 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:36.790180    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"]
Sep 10 23:27:36.790753 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:36.790444    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv" podUID="1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" containerName="maas-api" containerID="cri-o://5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373" gracePeriod=30
Sep 10 23:27:36.816434 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:36.816390    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:36.866059 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:36.866024    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"]
Sep 10 23:27:36.866317 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:36.866292    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8" podUID="7644980b-fc7f-4c80-9a27-24132523a407" containerName="payload-pre-processing" containerID="cri-o://b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279" gracePeriod=30
Sep 10 23:27:36.886943 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:36.886901    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:36.941433 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:36.941401    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.037117 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.037079    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.096556 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.096513    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.108316 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.108287    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:37.142100 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.142075    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-openshift-service-ca\") pod \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") "
Sep 10 23:27:37.142239 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.142180    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-metrics-certs\") pod \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") "
Sep 10 23:27:37.142239 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.142214    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2lw55\" (UniqueName: \"kubernetes.io/projected/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-kube-api-access-2lw55\") pod \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") "
Sep 10 23:27:37.142239 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.142230    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-maas-api-tls\") pod \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\" (UID: \"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3\") "
Sep 10 23:27:37.142885 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.142800    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" (UID: "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:37.144498 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.144476    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" (UID: "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:37.144597 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.144516    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" (UID: "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:27:37.144597 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.144532    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-kube-api-access-2lw55" (OuterVolumeSpecName: "kube-api-access-2lw55") pod "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" (UID: "1ffe5261-3aa5-4532-9a30-dee7e2d69fa3"). InnerVolumeSpecName "kube-api-access-2lw55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:37.150140 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.150121    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:37.243374 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.243340    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:27:37.243544 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.243395    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lw55\" (UniqueName: \"kubernetes.io/projected/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-kube-api-access-2lw55\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:27:37.243544 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.243422    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:27:37.243544 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.243434    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:27:37.344606 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.344495    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7644980b-fc7f-4c80-9a27-24132523a407-plugins-config-volume\") pod \"7644980b-fc7f-4c80-9a27-24132523a407\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") "
Sep 10 23:27:37.344606 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.344546    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nwjpk\" (UniqueName: \"kubernetes.io/projected/7644980b-fc7f-4c80-9a27-24132523a407-kube-api-access-nwjpk\") pod \"7644980b-fc7f-4c80-9a27-24132523a407\" (UID: \"7644980b-fc7f-4c80-9a27-24132523a407\") "
Sep 10 23:27:37.344948 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.344913    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7644980b-fc7f-4c80-9a27-24132523a407-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7644980b-fc7f-4c80-9a27-24132523a407" (UID: "7644980b-fc7f-4c80-9a27-24132523a407"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:27:37.346708 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.346687    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7644980b-fc7f-4c80-9a27-24132523a407-kube-api-access-nwjpk" (OuterVolumeSpecName: "kube-api-access-nwjpk") pod "7644980b-fc7f-4c80-9a27-24132523a407" (UID: "7644980b-fc7f-4c80-9a27-24132523a407"). InnerVolumeSpecName "kube-api-access-nwjpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:27:37.386404 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.386364    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:37.404480 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.404448    2577 generic.go:356] "Generic (PLEG): container finished" podID="1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" containerID="5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373" exitCode=0
Sep 10 23:27:37.404683 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.404489    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"
Sep 10 23:27:37.404683 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.404524    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv" event={"ID":"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3","Type":"ContainerDied","Data":"5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373"}
Sep 10 23:27:37.404683 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.404603    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv" event={"ID":"1ffe5261-3aa5-4532-9a30-dee7e2d69fa3","Type":"ContainerDied","Data":"baf8a4916708c84bc6844dea6c50b0be9ca9fbdb3641145207b5bb5c34264e65"}
Sep 10 23:27:37.404683 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.404638    2577 scope.go:117] "RemoveContainer" containerID="5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373"
Sep 10 23:27:37.406050 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.406027    2577 generic.go:356] "Generic (PLEG): container finished" podID="7644980b-fc7f-4c80-9a27-24132523a407" containerID="b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279" exitCode=0
Sep 10 23:27:37.406162 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.406069    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"
Sep 10 23:27:37.406162 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.406113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8" event={"ID":"7644980b-fc7f-4c80-9a27-24132523a407","Type":"ContainerDied","Data":"b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279"}
Sep 10 23:27:37.406252 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.406164    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8" event={"ID":"7644980b-fc7f-4c80-9a27-24132523a407","Type":"ContainerDied","Data":"01d822a66336615374278151180f4266092ed8e7da6f19b7152e14da8ccc5a4e"}
Sep 10 23:27:37.415071 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.415051    2577 scope.go:117] "RemoveContainer" containerID="5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373"
Sep 10 23:27:37.415437 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:27:37.415415    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373\": container with ID starting with 5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373 not found: ID does not exist" containerID="5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373"
Sep 10 23:27:37.415512 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.415447    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373"} err="failed to get container status \"5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373\": rpc error: code = NotFound desc = could not find container \"5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373\": container with ID starting with 5d2fcf944fca328efb491c8225014c0e45ba0df7c479c0748e86cb6c68fee373 not found: ID does not exist"
Sep 10 23:27:37.415512 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.415469    2577 scope.go:117] "RemoveContainer" containerID="b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279"
Sep 10 23:27:37.423719 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.423696    2577 scope.go:117] "RemoveContainer" containerID="b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279"
Sep 10 23:27:37.424029 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:27:37.424011    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279\": container with ID starting with b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279 not found: ID does not exist" containerID="b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279"
Sep 10 23:27:37.424101 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.424042    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279"} err="failed to get container status \"b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279\": rpc error: code = NotFound desc = could not find container \"b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279\": container with ID starting with b674ce050895e5f9a3676e7b776435c82bde827747753a73a5cb03453b23a279 not found: ID does not exist"
Sep 10 23:27:37.428243 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.428221    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"]
Sep 10 23:27:37.434562 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.434541    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-2cdd19-84d5445f7-qtclv"]
Sep 10 23:27:37.448419 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.448387    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7644980b-fc7f-4c80-9a27-24132523a407-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:27:37.448559 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.448429    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwjpk\" (UniqueName: \"kubernetes.io/projected/7644980b-fc7f-4c80-9a27-24132523a407-kube-api-access-nwjpk\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:27:37.450964 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.450936    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"]
Sep 10 23:27:37.453064 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:37.453043    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2cdd19-85cccbfc8-bw6l8"]
Sep 10 23:27:38.214805 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:38.214750    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" path="/var/lib/kubelet/pods/1ffe5261-3aa5-4532-9a30-dee7e2d69fa3/volumes"
Sep 10 23:27:38.215193 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:38.215164    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7644980b-fc7f-4c80-9a27-24132523a407" path="/var/lib/kubelet/pods/7644980b-fc7f-4c80-9a27-24132523a407/volumes"
Sep 10 23:27:43.647945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:43.647902    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:43.706442 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:43.706402    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:43.910763 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:43.910723    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:27:47.653008 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:27:47.652972    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:02.964017 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:02.963984    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:03.036044 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:03.036007    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:03.094971 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:03.094937    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:03.171048 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:03.171012    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:03.318348 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:03.318258    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:03.553518 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:03.553479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:08.628050 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:08.628004    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:31.860870 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:31.860827    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:45.567859 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:45.567822    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:45.833302 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:45.833205    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:51.922140 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:51.922097    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:54.602823 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:54.602790    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:28:59.910197 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:28:59.910153    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:06.918765 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:06.918724    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:08.140935 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:08.140898    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.010672 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.010638    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"]
Sep 10 23:29:18.011250 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.010928    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf" podUID="9b382f05-4dc8-4c24-9687-d0ccd6064a5a" containerName="maas-api" containerID="cri-o://ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a" gracePeriod=30
Sep 10 23:29:18.028200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.028162    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.114113 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.114074    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.229085 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.229039    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.266962 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.266895    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:29:18.288584 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.288545    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.449178 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.449135    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls\") pod \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") "
Sep 10 23:29:18.449382 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.449207    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-metrics-certs\") pod \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") "
Sep 10 23:29:18.449382 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.449257    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-openshift-service-ca\") pod \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") "
Sep 10 23:29:18.449491 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.449407    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7slfw\" (UniqueName: \"kubernetes.io/projected/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-kube-api-access-7slfw\") pod \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\" (UID: \"9b382f05-4dc8-4c24-9687-d0ccd6064a5a\") "
Sep 10 23:29:18.449665 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.449634    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9b382f05-4dc8-4c24-9687-d0ccd6064a5a" (UID: "9b382f05-4dc8-4c24-9687-d0ccd6064a5a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:18.449665 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.449818    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:18.451475 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.451444    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "9b382f05-4dc8-4c24-9687-d0ccd6064a5a" (UID: "9b382f05-4dc8-4c24-9687-d0ccd6064a5a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:18.451572 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.451473    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9b382f05-4dc8-4c24-9687-d0ccd6064a5a" (UID: "9b382f05-4dc8-4c24-9687-d0ccd6064a5a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:18.451572 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.451564    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-kube-api-access-7slfw" (OuterVolumeSpecName: "kube-api-access-7slfw") pod "9b382f05-4dc8-4c24-9687-d0ccd6064a5a" (UID: "9b382f05-4dc8-4c24-9687-d0ccd6064a5a"). InnerVolumeSpecName "kube-api-access-7slfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:18.550616 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.550501    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7slfw\" (UniqueName: \"kubernetes.io/projected/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-kube-api-access-7slfw\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:18.550616 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.550555    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:18.550616 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.550571    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9b382f05-4dc8-4c24-9687-d0ccd6064a5a-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:18.630732 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.630692    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.750059 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.750023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.816838 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.816729    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:18.894741 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.894702    2577 generic.go:356] "Generic (PLEG): container finished" podID="9b382f05-4dc8-4c24-9687-d0ccd6064a5a" containerID="ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a" exitCode=0
Sep 10 23:29:18.894984 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.894752    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"
Sep 10 23:29:18.894984 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.894810    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf" event={"ID":"9b382f05-4dc8-4c24-9687-d0ccd6064a5a","Type":"ContainerDied","Data":"ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a"}
Sep 10 23:29:18.894984 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.894873    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf" event={"ID":"9b382f05-4dc8-4c24-9687-d0ccd6064a5a","Type":"ContainerDied","Data":"1acc3afe45ecb90bd6ac54dc8f3ce43c7b6e5b2cc08d7540902b52cd4e4ba33d"}
Sep 10 23:29:18.894984 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.894904    2577 scope.go:117] "RemoveContainer" containerID="ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a"
Sep 10 23:29:18.904447 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.904425    2577 scope.go:117] "RemoveContainer" containerID="ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a"
Sep 10 23:29:18.904753 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:29:18.904731    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a\": container with ID starting with ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a not found: ID does not exist" containerID="ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a"
Sep 10 23:29:18.904881 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.904760    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a"} err="failed to get container status \"ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a\": rpc error: code = NotFound desc = could not find container \"ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a\": container with ID starting with ab1864bf7d1eba5ca0b241103b99a42c571715c2cd82c8dfcbdb6f1ecd987e1a not found: ID does not exist"
Sep 10 23:29:18.916707 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.916683    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"]
Sep 10 23:29:18.923503 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:18.923476    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-c6c079-856f7c95b6-569rf"]
Sep 10 23:29:20.214470 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:20.214439    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b382f05-4dc8-4c24-9687-d0ccd6064a5a" path="/var/lib/kubelet/pods/9b382f05-4dc8-4c24-9687-d0ccd6064a5a/volumes"
Sep 10 23:29:25.035976 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:25.035936    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:25.099081 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:25.099026    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:25.303465 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:25.303372    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:25.810437 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:25.810404    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:36.353141 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.353102    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"]
Sep 10 23:29:36.353617 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.353313    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" podUID="28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" containerName="maas-api" containerID="cri-o://216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763" gracePeriod=30
Sep 10 23:29:36.377251 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.377216    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:36.421812 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.421765    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"]
Sep 10 23:29:36.422114 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.422086    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn" podUID="9c666fb3-a182-4034-af8c-149c7a16f8b5" containerName="payload-pre-processing" containerID="cri-o://9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a" gracePeriod=30
Sep 10 23:29:36.448731 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.448695    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:36.511996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.511959    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:36.612850 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.612668    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:36.666318 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.666283    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:36.677130 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.677099    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:29:36.705753 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.705648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls\") pod \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") "
Sep 10 23:29:36.705946 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.705812    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-openshift-service-ca\") pod \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") "
Sep 10 23:29:36.705946 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.705882    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-metrics-certs\") pod \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") "
Sep 10 23:29:36.706063 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.705954    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldnqd\" (UniqueName: \"kubernetes.io/projected/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-kube-api-access-ldnqd\") pod \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\" (UID: \"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0\") "
Sep 10 23:29:36.706208 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.706180    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" (UID: "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:36.706395 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.706373    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:36.707724 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.707700    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:29:36.708345 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.708300    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-kube-api-access-ldnqd" (OuterVolumeSpecName: "kube-api-access-ldnqd") pod "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" (UID: "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0"). InnerVolumeSpecName "kube-api-access-ldnqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:36.708448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.708367    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" (UID: "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:36.708448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.708410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" (UID: "28f9b8b7-b116-4b4d-9416-fc7b597b5ff0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:36.807614 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.807575    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c666fb3-a182-4034-af8c-149c7a16f8b5-plugins-config-volume\") pod \"9c666fb3-a182-4034-af8c-149c7a16f8b5\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") "
Sep 10 23:29:36.807771 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.807623    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-68gzr\" (UniqueName: \"kubernetes.io/projected/9c666fb3-a182-4034-af8c-149c7a16f8b5-kube-api-access-68gzr\") pod \"9c666fb3-a182-4034-af8c-149c7a16f8b5\" (UID: \"9c666fb3-a182-4034-af8c-149c7a16f8b5\") "
Sep 10 23:29:36.808007 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.807986    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldnqd\" (UniqueName: \"kubernetes.io/projected/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-kube-api-access-ldnqd\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:36.808053 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.808014    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:36.808053 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.808028    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:36.808127 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.808051    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c666fb3-a182-4034-af8c-149c7a16f8b5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9c666fb3-a182-4034-af8c-149c7a16f8b5" (UID: "9c666fb3-a182-4034-af8c-149c7a16f8b5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:36.809945 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.809919    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c666fb3-a182-4034-af8c-149c7a16f8b5-kube-api-access-68gzr" (OuterVolumeSpecName: "kube-api-access-68gzr") pod "9c666fb3-a182-4034-af8c-149c7a16f8b5" (UID: "9c666fb3-a182-4034-af8c-149c7a16f8b5"). InnerVolumeSpecName "kube-api-access-68gzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:36.909278 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.909244    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c666fb3-a182-4034-af8c-149c7a16f8b5-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:36.909278 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.909273    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68gzr\" (UniqueName: \"kubernetes.io/projected/9c666fb3-a182-4034-af8c-149c7a16f8b5-kube-api-access-68gzr\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:36.979562 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.979531    2577 generic.go:356] "Generic (PLEG): container finished" podID="28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" containerID="216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763" exitCode=0
Sep 10 23:29:36.979772 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.979574    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"
Sep 10 23:29:36.979772 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.979617    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" event={"ID":"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0","Type":"ContainerDied","Data":"216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763"}
Sep 10 23:29:36.979772 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.979666    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w" event={"ID":"28f9b8b7-b116-4b4d-9416-fc7b597b5ff0","Type":"ContainerDied","Data":"b232710ba020c2b0a1c866911083e695d3f92b8d69eec86f45f38f4d9b0cf922"}
Sep 10 23:29:36.979772 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.979695    2577 scope.go:117] "RemoveContainer" containerID="216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763"
Sep 10 23:29:36.981155 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.981132    2577 generic.go:356] "Generic (PLEG): container finished" podID="9c666fb3-a182-4034-af8c-149c7a16f8b5" containerID="9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a" exitCode=0
Sep 10 23:29:36.981261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.981171    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"
Sep 10 23:29:36.981261 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.981214    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn" event={"ID":"9c666fb3-a182-4034-af8c-149c7a16f8b5","Type":"ContainerDied","Data":"9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a"}
Sep 10 23:29:36.981362 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.981264    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn" event={"ID":"9c666fb3-a182-4034-af8c-149c7a16f8b5","Type":"ContainerDied","Data":"1096ca5cc5df74faa483e1152052147d7e67c3d01bd0d2cdd424da25ef41e82f"}
Sep 10 23:29:36.989542 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.989524    2577 scope.go:117] "RemoveContainer" containerID="216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763"
Sep 10 23:29:36.989835 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:29:36.989811    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763\": container with ID starting with 216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763 not found: ID does not exist" containerID="216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763"
Sep 10 23:29:36.989889 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.989847    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763"} err="failed to get container status \"216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763\": rpc error: code = NotFound desc = could not find container \"216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763\": container with ID starting with 216de597b2372773f802c23b9433f6ed36c587f3e37c45f85ba8add604841763 not found: ID does not exist"
Sep 10 23:29:36.989889 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.989871    2577 scope.go:117] "RemoveContainer" containerID="9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a"
Sep 10 23:29:36.997610 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.997589    2577 scope.go:117] "RemoveContainer" containerID="9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a"
Sep 10 23:29:36.998135 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:29:36.998053    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a\": container with ID starting with 9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a not found: ID does not exist" containerID="9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a"
Sep 10 23:29:36.998135 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:36.998086    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a"} err="failed to get container status \"9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a\": rpc error: code = NotFound desc = could not find container \"9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a\": container with ID starting with 9f723e9232d77cb0aa0caa1c15267d60c9473df31b643fa1c25a50b72954460a not found: ID does not exist"
Sep 10 23:29:37.004332 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:37.004306    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"]
Sep 10 23:29:37.006289 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:37.006266    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-2da572-774f7b65f9-lq22w"]
Sep 10 23:29:37.014702 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:37.014679    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:37.018859 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:37.018839    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"]
Sep 10 23:29:37.021769 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:37.021748    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-2da572-d9477b454-q9qtn"]
Sep 10 23:29:38.216249 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:38.216215    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" path="/var/lib/kubelet/pods/28f9b8b7-b116-4b4d-9416-fc7b597b5ff0/volumes"
Sep 10 23:29:38.216626 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:38.216573    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c666fb3-a182-4034-af8c-149c7a16f8b5" path="/var/lib/kubelet/pods/9c666fb3-a182-4034-af8c-149c7a16f8b5/volumes"
Sep 10 23:29:43.494060 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:43.494000    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:43.593955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:43.593912    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:43.774351 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:43.774235    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:44.205061 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:44.205023    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:54.926056 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:54.926027    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"]
Sep 10 23:29:54.926419 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:54.926214    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" podUID="cf228744-5cd6-4bf6-bf78-d67a476f2dd2" containerName="maas-api" containerID="cri-o://65d02fe2e922ac8c409db4d766e1a4de27996b7fd1a2efd492f041e5e62ad247" gracePeriod=30
Sep 10 23:29:54.979293 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:54.979256    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:55.042715 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.042666    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"]
Sep 10 23:29:55.043052 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.042996    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87" podUID="7047ce95-0fa3-45a1-ab15-6aecbf49c32f" containerName="payload-pre-processing" containerID="cri-o://8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306" gracePeriod=30
Sep 10 23:29:55.044529 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.044504    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"]
Sep 10 23:29:55.044731 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.044703    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5" podUID="e78b32c4-c119-482c-9f62-dfc4f2ce428f" containerName="payload-processing" containerID="cri-o://e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e" gracePeriod=30
Sep 10 23:29:55.072087 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.072056    2577 generic.go:356] "Generic (PLEG): container finished" podID="cf228744-5cd6-4bf6-bf78-d67a476f2dd2" containerID="65d02fe2e922ac8c409db4d766e1a4de27996b7fd1a2efd492f041e5e62ad247" exitCode=0
Sep 10 23:29:55.072205 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.072163    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" event={"ID":"cf228744-5cd6-4bf6-bf78-d67a476f2dd2","Type":"ContainerDied","Data":"65d02fe2e922ac8c409db4d766e1a4de27996b7fd1a2efd492f041e5e62ad247"}
Sep 10 23:29:55.080906 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.080873    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:55.177283 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.177219    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:55.188674 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.188640    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:29:55.269041 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.269016    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls\") pod \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") "
Sep 10 23:29:55.269255 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.269106    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-metrics-certs\") pod \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") "
Sep 10 23:29:55.269255 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.269184    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-openshift-service-ca\") pod \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") "
Sep 10 23:29:55.269255 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.269223    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jjkpg\" (UniqueName: \"kubernetes.io/projected/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-kube-api-access-jjkpg\") pod \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\" (UID: \"cf228744-5cd6-4bf6-bf78-d67a476f2dd2\") "
Sep 10 23:29:55.270435 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.270407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cf228744-5cd6-4bf6-bf78-d67a476f2dd2" (UID: "cf228744-5cd6-4bf6-bf78-d67a476f2dd2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:55.271866 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.271835    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cf228744-5cd6-4bf6-bf78-d67a476f2dd2" (UID: "cf228744-5cd6-4bf6-bf78-d67a476f2dd2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:55.272227 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.272200    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cf228744-5cd6-4bf6-bf78-d67a476f2dd2" (UID: "cf228744-5cd6-4bf6-bf78-d67a476f2dd2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:29:55.272675 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.272655    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-kube-api-access-jjkpg" (OuterVolumeSpecName: "kube-api-access-jjkpg") pod "cf228744-5cd6-4bf6-bf78-d67a476f2dd2" (UID: "cf228744-5cd6-4bf6-bf78-d67a476f2dd2"). InnerVolumeSpecName "kube-api-access-jjkpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:55.286995 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.286971    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:29:55.317999 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.317975    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:29:55.362165 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.362133    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:55.370162 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.370138    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdkwc\" (UniqueName: \"kubernetes.io/projected/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-kube-api-access-qdkwc\") pod \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") "
Sep 10 23:29:55.370308 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.370205    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-plugins-config-volume\") pod \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\" (UID: \"7047ce95-0fa3-45a1-ab15-6aecbf49c32f\") "
Sep 10 23:29:55.370404 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.370395    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjkpg\" (UniqueName: \"kubernetes.io/projected/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-kube-api-access-jjkpg\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.370440 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.370407    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.370440 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.370416    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.370440 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.370423    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf228744-5cd6-4bf6-bf78-d67a476f2dd2-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.370604 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.370575    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7047ce95-0fa3-45a1-ab15-6aecbf49c32f" (UID: "7047ce95-0fa3-45a1-ab15-6aecbf49c32f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:55.372229 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.372209    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-kube-api-access-qdkwc" (OuterVolumeSpecName: "kube-api-access-qdkwc") pod "7047ce95-0fa3-45a1-ab15-6aecbf49c32f" (UID: "7047ce95-0fa3-45a1-ab15-6aecbf49c32f"). InnerVolumeSpecName "kube-api-access-qdkwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:55.464237 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.464143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:55.471296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.471263    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b32c4-c119-482c-9f62-dfc4f2ce428f-plugins-config-volume\") pod \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") "
Sep 10 23:29:55.471433 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.471402    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5sfq\" (UniqueName: \"kubernetes.io/projected/e78b32c4-c119-482c-9f62-dfc4f2ce428f-kube-api-access-t5sfq\") pod \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\" (UID: \"e78b32c4-c119-482c-9f62-dfc4f2ce428f\") "
Sep 10 23:29:55.471630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.471614    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdkwc\" (UniqueName: \"kubernetes.io/projected/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-kube-api-access-qdkwc\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.471683 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.471635    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7047ce95-0fa3-45a1-ab15-6aecbf49c32f-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.471683 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.471637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78b32c4-c119-482c-9f62-dfc4f2ce428f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e78b32c4-c119-482c-9f62-dfc4f2ce428f" (UID: "e78b32c4-c119-482c-9f62-dfc4f2ce428f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:29:55.473510 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.473483    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e78b32c4-c119-482c-9f62-dfc4f2ce428f-kube-api-access-t5sfq" (OuterVolumeSpecName: "kube-api-access-t5sfq") pod "e78b32c4-c119-482c-9f62-dfc4f2ce428f" (UID: "e78b32c4-c119-482c-9f62-dfc4f2ce428f"). InnerVolumeSpecName "kube-api-access-t5sfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:29:55.573059 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.573021    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5sfq\" (UniqueName: \"kubernetes.io/projected/e78b32c4-c119-482c-9f62-dfc4f2ce428f-kube-api-access-t5sfq\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.573059 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.573053    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e78b32c4-c119-482c-9f62-dfc4f2ce428f-plugins-config-volume\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:29:55.860277 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:55.860185    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:29:56.078479 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.078445    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"
Sep 10 23:29:56.078953 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.078444    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc" event={"ID":"cf228744-5cd6-4bf6-bf78-d67a476f2dd2","Type":"ContainerDied","Data":"380ee6e895f524b9cb72a2d9ca5cd421c972e09b52875a327c9ee3071fb9d522"}
Sep 10 23:29:56.078953 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.078573    2577 scope.go:117] "RemoveContainer" containerID="65d02fe2e922ac8c409db4d766e1a4de27996b7fd1a2efd492f041e5e62ad247"
Sep 10 23:29:56.079996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.079698    2577 generic.go:356] "Generic (PLEG): container finished" podID="7047ce95-0fa3-45a1-ab15-6aecbf49c32f" containerID="8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306" exitCode=0
Sep 10 23:29:56.079996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.079743    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"
Sep 10 23:29:56.079996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.079799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87" event={"ID":"7047ce95-0fa3-45a1-ab15-6aecbf49c32f","Type":"ContainerDied","Data":"8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306"}
Sep 10 23:29:56.079996 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.079847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87" event={"ID":"7047ce95-0fa3-45a1-ab15-6aecbf49c32f","Type":"ContainerDied","Data":"2b90e71f9155487ea9de868bfa7cfeac18023f0b8efe1d55653779928d4ce2f9"}
Sep 10 23:29:56.081295 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.081270    2577 generic.go:356] "Generic (PLEG): container finished" podID="e78b32c4-c119-482c-9f62-dfc4f2ce428f" containerID="e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e" exitCode=0
Sep 10 23:29:56.081380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.081313    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"
Sep 10 23:29:56.081380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.081351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5" event={"ID":"e78b32c4-c119-482c-9f62-dfc4f2ce428f","Type":"ContainerDied","Data":"e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e"}
Sep 10 23:29:56.081459 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.081407    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5" event={"ID":"e78b32c4-c119-482c-9f62-dfc4f2ce428f","Type":"ContainerDied","Data":"e552d68666550510f270d8d9473e09584b3d2bb5e6c38e726f86ec8866d47740"}
Sep 10 23:29:56.087815 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.087769    2577 scope.go:117] "RemoveContainer" containerID="8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306"
Sep 10 23:29:56.096122 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.096105    2577 scope.go:117] "RemoveContainer" containerID="8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306"
Sep 10 23:29:56.096356 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:29:56.096333    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306\": container with ID starting with 8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306 not found: ID does not exist" containerID="8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306"
Sep 10 23:29:56.096424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.096367    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306"} err="failed to get container status \"8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306\": rpc error: code = NotFound desc = could not find container \"8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306\": container with ID starting with 8c5e6c49fb04f3736e9d39f32be15c613ddf6a79c2da8b00e5d14a1daf0b5306 not found: ID does not exist"
Sep 10 23:29:56.096424 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.096388    2577 scope.go:117] "RemoveContainer" containerID="e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e"
Sep 10 23:29:56.103675 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.103657    2577 scope.go:117] "RemoveContainer" containerID="e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e"
Sep 10 23:29:56.103928 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:29:56.103910    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e\": container with ID starting with e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e not found: ID does not exist" containerID="e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e"
Sep 10 23:29:56.103991 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.103938    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e"} err="failed to get container status \"e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e\": rpc error: code = NotFound desc = could not find container \"e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e\": container with ID starting with e233faec3781d4183b6bc32e3de85fd1706520c1d90274bae2b53b759ee8423e not found: ID does not exist"
Sep 10 23:29:56.110036 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.110016    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"]
Sep 10 23:29:56.115902 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.115856    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-4b268f-6c4f46685d-pnh87"]
Sep 10 23:29:56.129269 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.129247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"]
Sep 10 23:29:56.135128 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.135105    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-4b268f-999d49f55-6d7dc"]
Sep 10 23:29:56.147846 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.147828    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"]
Sep 10 23:29:56.150406 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.150387    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-4b268f-d9c798f57-55gq5"]
Sep 10 23:29:56.213986 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.213961    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7047ce95-0fa3-45a1-ab15-6aecbf49c32f" path="/var/lib/kubelet/pods/7047ce95-0fa3-45a1-ab15-6aecbf49c32f/volumes"
Sep 10 23:29:56.214350 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.214338    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf228744-5cd6-4bf6-bf78-d67a476f2dd2" path="/var/lib/kubelet/pods/cf228744-5cd6-4bf6-bf78-d67a476f2dd2/volumes"
Sep 10 23:29:56.214637 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:29:56.214626    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78b32c4-c119-482c-9f62-dfc4f2ce428f" path="/var/lib/kubelet/pods/e78b32c4-c119-482c-9f62-dfc4f2ce428f/volumes"
Sep 10 23:30:03.709490 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:03.709457    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:30:04.145079 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:04.144997    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:30:04.207347 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:04.207321    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:30:04.527769 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:04.527728    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:30:07.039081 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.039047    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cf685499f-bjvqh"]
Sep 10 23:30:07.039518 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.039229    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cf685499f-bjvqh" podUID="7af435e9-16a6-4feb-8649-3ddf1af20c8d" containerName="manager" containerID="cri-o://68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561" gracePeriod=30
Sep 10 23:30:07.276595 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.276570    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:30:07.371425 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.371334    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4frj9\" (UniqueName: \"kubernetes.io/projected/7af435e9-16a6-4feb-8649-3ddf1af20c8d-kube-api-access-4frj9\") pod \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") "
Sep 10 23:30:07.371425 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.371376    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-metrics-certs\") pod \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") "
Sep 10 23:30:07.371624 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.371493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-webhook-cert\") pod \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\" (UID: \"7af435e9-16a6-4feb-8649-3ddf1af20c8d\") "
Sep 10 23:30:07.373413 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.373375    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7af435e9-16a6-4feb-8649-3ddf1af20c8d" (UID: "7af435e9-16a6-4feb-8649-3ddf1af20c8d"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:30:07.373545 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.373407    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7af435e9-16a6-4feb-8649-3ddf1af20c8d" (UID: "7af435e9-16a6-4feb-8649-3ddf1af20c8d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:30:07.373545 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.373464    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7af435e9-16a6-4feb-8649-3ddf1af20c8d-kube-api-access-4frj9" (OuterVolumeSpecName: "kube-api-access-4frj9") pod "7af435e9-16a6-4feb-8649-3ddf1af20c8d" (UID: "7af435e9-16a6-4feb-8649-3ddf1af20c8d"). InnerVolumeSpecName "kube-api-access-4frj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:30:07.472299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.472261    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4frj9\" (UniqueName: \"kubernetes.io/projected/7af435e9-16a6-4feb-8649-3ddf1af20c8d-kube-api-access-4frj9\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:30:07.472299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.472293    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:30:07.472299 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:07.472302    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7af435e9-16a6-4feb-8649-3ddf1af20c8d-webhook-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:30:08.137965 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.137929    2577 generic.go:356] "Generic (PLEG): container finished" podID="7af435e9-16a6-4feb-8649-3ddf1af20c8d" containerID="68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561" exitCode=0
Sep 10 23:30:08.138431 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.137971    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-bjvqh"
Sep 10 23:30:08.138431 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.138014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-bjvqh" event={"ID":"7af435e9-16a6-4feb-8649-3ddf1af20c8d","Type":"ContainerDied","Data":"68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561"}
Sep 10 23:30:08.138431 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.138064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-bjvqh" event={"ID":"7af435e9-16a6-4feb-8649-3ddf1af20c8d","Type":"ContainerDied","Data":"3ee0c12de83855765919c8c2d9dcfabcb1956e3a33c8d6e18135b09ecb585213"}
Sep 10 23:30:08.138431 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.138086    2577 scope.go:117] "RemoveContainer" containerID="68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561"
Sep 10 23:30:08.147587 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.147566    2577 scope.go:117] "RemoveContainer" containerID="68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561"
Sep 10 23:30:08.147850 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:30:08.147832    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561\": container with ID starting with 68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561 not found: ID does not exist" containerID="68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561"
Sep 10 23:30:08.147896 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.147858    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561"} err="failed to get container status \"68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561\": rpc error: code = NotFound desc = could not find container \"68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561\": container with ID starting with 68220a354dc898924d7801951108553e21f2e9aea7d202d7a366b856712d5561 not found: ID does not exist"
Sep 10 23:30:08.182543 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.182517    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cf685499f-bjvqh"]
Sep 10 23:30:08.184641 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.184622    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cf685499f-bjvqh"]
Sep 10 23:30:08.213600 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:30:08.213579    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7af435e9-16a6-4feb-8649-3ddf1af20c8d" path="/var/lib/kubelet/pods/7af435e9-16a6-4feb-8649-3ddf1af20c8d/volumes"
Sep 10 23:31:09.016268 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016233    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cf685499f-zndrn"]
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016589    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78b32c4-c119-482c-9f62-dfc4f2ce428f" containerName="payload-processing"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016600    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78b32c4-c119-482c-9f62-dfc4f2ce428f" containerName="payload-processing"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016610    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7af435e9-16a6-4feb-8649-3ddf1af20c8d" containerName="manager"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016617    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7af435e9-16a6-4feb-8649-3ddf1af20c8d" containerName="manager"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016625    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b382f05-4dc8-4c24-9687-d0ccd6064a5a" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016631    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b382f05-4dc8-4c24-9687-d0ccd6064a5a" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016638    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7047ce95-0fa3-45a1-ab15-6aecbf49c32f" containerName="payload-pre-processing"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016644    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7047ce95-0fa3-45a1-ab15-6aecbf49c32f" containerName="payload-pre-processing"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016652    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c666fb3-a182-4034-af8c-149c7a16f8b5" containerName="payload-pre-processing"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016656    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c666fb3-a182-4034-af8c-149c7a16f8b5" containerName="payload-pre-processing"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016663    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf228744-5cd6-4bf6-bf78-d67a476f2dd2" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016668    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf228744-5cd6-4bf6-bf78-d67a476f2dd2" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016675    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016680    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016685    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016690    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" containerName="maas-api"
Sep 10 23:31:09.016687 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016696    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7644980b-fc7f-4c80-9a27-24132523a407" containerName="payload-pre-processing"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016701    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7644980b-fc7f-4c80-9a27-24132523a407" containerName="payload-pre-processing"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016752    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c666fb3-a182-4034-af8c-149c7a16f8b5" containerName="payload-pre-processing"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016759    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ffe5261-3aa5-4532-9a30-dee7e2d69fa3" containerName="maas-api"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016765    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7af435e9-16a6-4feb-8649-3ddf1af20c8d" containerName="manager"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016771    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="28f9b8b7-b116-4b4d-9416-fc7b597b5ff0" containerName="maas-api"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016790    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b382f05-4dc8-4c24-9687-d0ccd6064a5a" containerName="maas-api"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016799    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78b32c4-c119-482c-9f62-dfc4f2ce428f" containerName="payload-processing"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016809    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7047ce95-0fa3-45a1-ab15-6aecbf49c32f" containerName="payload-pre-processing"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016816    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7644980b-fc7f-4c80-9a27-24132523a407" containerName="payload-pre-processing"
Sep 10 23:31:09.017234 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.016824    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf228744-5cd6-4bf6-bf78-d67a476f2dd2" containerName="maas-api"
Sep 10 23:31:09.019639 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.019624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.022902 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.022878    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2npq7\""
Sep 10 23:31:09.023043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.022929    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 23:31:09.023043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.022888    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 23:31:09.023043 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.022887    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 23:31:09.031386 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.031362    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cf685499f-zndrn"]
Sep 10 23:31:09.097247 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.097213    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-metrics-certs\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.097419 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.097257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-webhook-cert\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.097419 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.097291    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gckcn\" (UniqueName: \"kubernetes.io/projected/d469de89-7a51-45d5-aaa1-966e93b60335-kube-api-access-gckcn\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.198232 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.198201    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-metrics-certs\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.198427 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.198239    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-webhook-cert\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.198427 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.198358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gckcn\" (UniqueName: \"kubernetes.io/projected/d469de89-7a51-45d5-aaa1-966e93b60335-kube-api-access-gckcn\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.200692 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.200669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-metrics-certs\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.200817 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.200699    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-webhook-cert\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.208595 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.208569    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gckcn\" (UniqueName: \"kubernetes.io/projected/d469de89-7a51-45d5-aaa1-966e93b60335-kube-api-access-gckcn\") pod \"maas-controller-cf685499f-zndrn\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") " pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.329638 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.329541    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:09.602983 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.602906    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cf685499f-zndrn"]
Sep 10 23:31:09.605927 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:31:09.605895    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd469de89_7a51_45d5_aaa1_966e93b60335.slice/crio-f1713e99e592e24bb21b4a75534ba2e8f5a5f52d48d0361cdeec84207b5606c8 WatchSource:0}: Error finding container f1713e99e592e24bb21b4a75534ba2e8f5a5f52d48d0361cdeec84207b5606c8: Status 404 returned error can't find the container with id f1713e99e592e24bb21b4a75534ba2e8f5a5f52d48d0361cdeec84207b5606c8
Sep 10 23:31:09.607228 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:09.607210    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 23:31:10.418183 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:10.418145    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-zndrn" event={"ID":"d469de89-7a51-45d5-aaa1-966e93b60335","Type":"ContainerStarted","Data":"6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c"}
Sep 10 23:31:10.418632 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:10.418210    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:10.418632 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:10.418232    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-zndrn" event={"ID":"d469de89-7a51-45d5-aaa1-966e93b60335","Type":"ContainerStarted","Data":"f1713e99e592e24bb21b4a75534ba2e8f5a5f52d48d0361cdeec84207b5606c8"}
Sep 10 23:31:10.440099 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:10.440047    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cf685499f-zndrn" podStartSLOduration=1.986453807 podStartE2EDuration="2.440030046s" totalImagesPullingTime="453.576239ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:31:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:31:10.438404116 +0000 UTC m=+1986.874458901" watchObservedRunningTime="2026-09-10 23:31:10.440030046 +0000 UTC m=+1986.876084817"
Sep 10 23:31:21.425210 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:21.425177    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:28.492217 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:28.492184    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:30.633643 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:30.633607    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:30.742630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:30.742589    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cf685499f-zndrn"]
Sep 10 23:31:30.742879 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:30.742845    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-cf685499f-zndrn" podUID="d469de89-7a51-45d5-aaa1-966e93b60335" containerName="manager" containerID="cri-o://6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c" gracePeriod=30
Sep 10 23:31:30.987118 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:30.987093    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:31.090492 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.090462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-webhook-cert\") pod \"d469de89-7a51-45d5-aaa1-966e93b60335\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") "
Sep 10 23:31:31.090492 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.090504    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-metrics-certs\") pod \"d469de89-7a51-45d5-aaa1-966e93b60335\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") "
Sep 10 23:31:31.090745 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.090608    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gckcn\" (UniqueName: \"kubernetes.io/projected/d469de89-7a51-45d5-aaa1-966e93b60335-kube-api-access-gckcn\") pod \"d469de89-7a51-45d5-aaa1-966e93b60335\" (UID: \"d469de89-7a51-45d5-aaa1-966e93b60335\") "
Sep 10 23:31:31.092551 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.092520    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "d469de89-7a51-45d5-aaa1-966e93b60335" (UID: "d469de89-7a51-45d5-aaa1-966e93b60335"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:31:31.092671 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.092581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d469de89-7a51-45d5-aaa1-966e93b60335-kube-api-access-gckcn" (OuterVolumeSpecName: "kube-api-access-gckcn") pod "d469de89-7a51-45d5-aaa1-966e93b60335" (UID: "d469de89-7a51-45d5-aaa1-966e93b60335"). InnerVolumeSpecName "kube-api-access-gckcn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:31:31.092671 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.092594    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "d469de89-7a51-45d5-aaa1-966e93b60335" (UID: "d469de89-7a51-45d5-aaa1-966e93b60335"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:31:31.192291 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.192213    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gckcn\" (UniqueName: \"kubernetes.io/projected/d469de89-7a51-45d5-aaa1-966e93b60335-kube-api-access-gckcn\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:31:31.192291 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.192241    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-webhook-cert\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:31:31.192291 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.192250    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d469de89-7a51-45d5-aaa1-966e93b60335-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:31:31.511956 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.511867    2577 generic.go:356] "Generic (PLEG): container finished" podID="d469de89-7a51-45d5-aaa1-966e93b60335" containerID="6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c" exitCode=0
Sep 10 23:31:31.511956 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.511910    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-zndrn"
Sep 10 23:31:31.512134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.511957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-zndrn" event={"ID":"d469de89-7a51-45d5-aaa1-966e93b60335","Type":"ContainerDied","Data":"6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c"}
Sep 10 23:31:31.512134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.512006    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-zndrn" event={"ID":"d469de89-7a51-45d5-aaa1-966e93b60335","Type":"ContainerDied","Data":"f1713e99e592e24bb21b4a75534ba2e8f5a5f52d48d0361cdeec84207b5606c8"}
Sep 10 23:31:31.512134 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.512031    2577 scope.go:117] "RemoveContainer" containerID="6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c"
Sep 10 23:31:31.520892 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.520872    2577 scope.go:117] "RemoveContainer" containerID="6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c"
Sep 10 23:31:31.521162 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:31:31.521144    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c\": container with ID starting with 6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c not found: ID does not exist" containerID="6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c"
Sep 10 23:31:31.521207 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.521172    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c"} err="failed to get container status \"6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c\": rpc error: code = NotFound desc = could not find container \"6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c\": container with ID starting with 6e49745e0786276e2f9b18725feb75d425c5a3ba409554feac5c06378739718c not found: ID does not exist"
Sep 10 23:31:31.534673 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.534648    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-cf685499f-zndrn"]
Sep 10 23:31:31.539707 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:31.539684    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-cf685499f-zndrn"]
Sep 10 23:31:32.213477 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.213442    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:31:32.214213 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.214193    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d469de89-7a51-45d5-aaa1-966e93b60335" path="/var/lib/kubelet/pods/d469de89-7a51-45d5-aaa1-966e93b60335/volumes"
Sep 10 23:31:32.337710 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.337676    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-cf685499f-9dnbp"]
Sep 10 23:31:32.338020 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.338006    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d469de89-7a51-45d5-aaa1-966e93b60335" containerName="manager"
Sep 10 23:31:32.338065 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.338021    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d469de89-7a51-45d5-aaa1-966e93b60335" containerName="manager"
Sep 10 23:31:32.338099 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.338086    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d469de89-7a51-45d5-aaa1-966e93b60335" containerName="manager"
Sep 10 23:31:32.342199 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.342176    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.345330 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.345309    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 23:31:32.345474 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.345332    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-2npq7\""
Sep 10 23:31:32.345474 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.345355    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 23:31:32.345474 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.345401    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 23:31:32.350552 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.350525    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cf685499f-9dnbp"]
Sep 10 23:31:32.503719 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.503627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qp6dk\" (UniqueName: \"kubernetes.io/projected/85148d71-b42f-42c9-9bcb-b69558766d02-kube-api-access-qp6dk\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.503895 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.503729    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85148d71-b42f-42c9-9bcb-b69558766d02-metrics-certs\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.503895 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.503839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85148d71-b42f-42c9-9bcb-b69558766d02-webhook-cert\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.604527 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.604477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85148d71-b42f-42c9-9bcb-b69558766d02-metrics-certs\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.604711 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.604562    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85148d71-b42f-42c9-9bcb-b69558766d02-webhook-cert\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.604711 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.604592    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qp6dk\" (UniqueName: \"kubernetes.io/projected/85148d71-b42f-42c9-9bcb-b69558766d02-kube-api-access-qp6dk\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.607037 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.607013    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/85148d71-b42f-42c9-9bcb-b69558766d02-metrics-certs\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.607142 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.607065    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/85148d71-b42f-42c9-9bcb-b69558766d02-webhook-cert\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.613683 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.613657    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qp6dk\" (UniqueName: \"kubernetes.io/projected/85148d71-b42f-42c9-9bcb-b69558766d02-kube-api-access-qp6dk\") pod \"maas-controller-cf685499f-9dnbp\" (UID: \"85148d71-b42f-42c9-9bcb-b69558766d02\") " pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.654213 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.654178    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:32.924809 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:32.924769    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-cf685499f-9dnbp"]
Sep 10 23:31:32.926875 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:31:32.926844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85148d71_b42f_42c9_9bcb_b69558766d02.slice/crio-173860066e7fa0cfdb41ac0a246b7fd03c93fb64da3d1fd2430ad3760b0ae522 WatchSource:0}: Error finding container 173860066e7fa0cfdb41ac0a246b7fd03c93fb64da3d1fd2430ad3760b0ae522: Status 404 returned error can't find the container with id 173860066e7fa0cfdb41ac0a246b7fd03c93fb64da3d1fd2430ad3760b0ae522
Sep 10 23:31:33.524451 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:33.524353    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-9dnbp" event={"ID":"85148d71-b42f-42c9-9bcb-b69558766d02","Type":"ContainerStarted","Data":"29b7a50dc8293a61c9d1fc54524f3f0e70c247ce72264cc985f49ce9c1441e25"}
Sep 10 23:31:33.524451 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:33.524433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:31:33.524872 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:33.524460    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-cf685499f-9dnbp" event={"ID":"85148d71-b42f-42c9-9bcb-b69558766d02","Type":"ContainerStarted","Data":"173860066e7fa0cfdb41ac0a246b7fd03c93fb64da3d1fd2430ad3760b0ae522"}
Sep 10 23:31:33.542774 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:33.542721    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-cf685499f-9dnbp" podStartSLOduration=1.208456768 podStartE2EDuration="1.542705s" totalImagesPullingTime="334.248232ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:31:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:31:33.540572567 +0000 UTC m=+2009.976627339" watchObservedRunningTime="2026-09-10 23:31:33.542705 +0000 UTC m=+2009.978759876"
Sep 10 23:31:44.531802 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:31:44.531742    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-cf685499f-9dnbp"
Sep 10 23:34:22.209470 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:34:22.209371    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:37.124936 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:34:37.124905    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:34:44.500194 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:34:44.500159    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:12.512195 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:12.512158    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:12.613200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:12.613164    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:20.040810 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:20.040765    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:20.772988 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:20.772949    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:23.861514 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:23.861479    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:23.962448 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:23.962413    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:35:37.516365 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.516277    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-p29cf"]
Sep 10 23:35:37.520441 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.520409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.531980 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.531950    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-p29cf"]
Sep 10 23:35:37.571746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.571700    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s4p2q\" (UniqueName: \"kubernetes.io/projected/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-kube-api-access-s4p2q\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.571746 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.571750    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-metrics-certs\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.571995 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.571862    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-openshift-service-ca\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.571995 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.571902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-maas-api-tls\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.673163 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.673123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-openshift-service-ca\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.673387 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.673181    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-maas-api-tls\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.673387 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.673234    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s4p2q\" (UniqueName: \"kubernetes.io/projected/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-kube-api-access-s4p2q\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.673387 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.673265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-metrics-certs\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.673812 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.673768    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-openshift-service-ca\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.675810 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.675767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-metrics-certs\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.675908 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.675842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-maas-api-tls\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.682712 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.682673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s4p2q\" (UniqueName: \"kubernetes.io/projected/b0fd6cba-336c-4cf8-8e98-8aae6e907fc3-kube-api-access-s4p2q\") pod \"maas-api-559477f47-p29cf\" (UID: \"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3\") " pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:37.834086 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:37.833992    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:38.106741 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:38.106654    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-559477f47-p29cf"]
Sep 10 23:35:38.110590 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:35:38.110557    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0fd6cba_336c_4cf8_8e98_8aae6e907fc3.slice/crio-92b345544c9047ee691273b49a1d56af3b45a40fe886ad97de84a765c1d3df17 WatchSource:0}: Error finding container 92b345544c9047ee691273b49a1d56af3b45a40fe886ad97de84a765c1d3df17: Status 404 returned error can't find the container with id 92b345544c9047ee691273b49a1d56af3b45a40fe886ad97de84a765c1d3df17
Sep 10 23:35:38.625431 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:38.625389    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf" event={"ID":"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3","Type":"ContainerStarted","Data":"92b345544c9047ee691273b49a1d56af3b45a40fe886ad97de84a765c1d3df17"}
Sep 10 23:35:39.631937 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:39.631893    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf" event={"ID":"b0fd6cba-336c-4cf8-8e98-8aae6e907fc3","Type":"ContainerStarted","Data":"63b44489775eded372127bbdf78c3835c4d571b13945e21703bc34c051e0070e"}
Sep 10 23:35:39.632326 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:39.632027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:39.649586 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:39.649532    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf" podStartSLOduration=2.057783134 podStartE2EDuration="2.649515049s" totalImagesPullingTime="591.731915ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 23:35:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:35:39.648191383 +0000 UTC m=+2256.084246146" watchObservedRunningTime="2026-09-10 23:35:39.649515049 +0000 UTC m=+2256.085569821"
Sep 10 23:35:45.642764 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:45.642735    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-559477f47-p29cf"
Sep 10 23:35:45.685282 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:45.685247    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"]
Sep 10 23:35:45.685707 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:45.685515    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg" podUID="110f6dd0-4140-4c6c-9653-9b5d5415bb16" containerName="maas-api" containerID="cri-o://6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64" gracePeriod=30
Sep 10 23:35:46.932576 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.932543    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:35:46.957716 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.957685    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110f6dd0-4140-4c6c-9653-9b5d5415bb16-openshift-service-ca\") pod \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") "
Sep 10 23:35:46.957716 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.957727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-maas-api-tls\") pod \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") "
Sep 10 23:35:46.957955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.957832    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-metrics-certs\") pod \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") "
Sep 10 23:35:46.957955 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.957885    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxrzt\" (UniqueName: \"kubernetes.io/projected/110f6dd0-4140-4c6c-9653-9b5d5415bb16-kube-api-access-vxrzt\") pod \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\" (UID: \"110f6dd0-4140-4c6c-9653-9b5d5415bb16\") "
Sep 10 23:35:46.958149 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.958116    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110f6dd0-4140-4c6c-9653-9b5d5415bb16-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "110f6dd0-4140-4c6c-9653-9b5d5415bb16" (UID: "110f6dd0-4140-4c6c-9653-9b5d5415bb16"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 23:35:46.959969 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.959940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "110f6dd0-4140-4c6c-9653-9b5d5415bb16" (UID: "110f6dd0-4140-4c6c-9653-9b5d5415bb16"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:46.960078 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.959996    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/110f6dd0-4140-4c6c-9653-9b5d5415bb16-kube-api-access-vxrzt" (OuterVolumeSpecName: "kube-api-access-vxrzt") pod "110f6dd0-4140-4c6c-9653-9b5d5415bb16" (UID: "110f6dd0-4140-4c6c-9653-9b5d5415bb16"). InnerVolumeSpecName "kube-api-access-vxrzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 23:35:46.960138 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:46.960099    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "110f6dd0-4140-4c6c-9653-9b5d5415bb16" (UID: "110f6dd0-4140-4c6c-9653-9b5d5415bb16"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 23:35:47.059305 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.059223    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-metrics-certs\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:35:47.059305 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.059251    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxrzt\" (UniqueName: \"kubernetes.io/projected/110f6dd0-4140-4c6c-9653-9b5d5415bb16-kube-api-access-vxrzt\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:35:47.059305 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.059261    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/110f6dd0-4140-4c6c-9653-9b5d5415bb16-openshift-service-ca\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:35:47.059305 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.059269    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/110f6dd0-4140-4c6c-9653-9b5d5415bb16-maas-api-tls\") on node \"ip-10-0-138-150.ec2.internal\" DevicePath \"\""
Sep 10 23:35:47.669289 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.669253    2577 generic.go:356] "Generic (PLEG): container finished" podID="110f6dd0-4140-4c6c-9653-9b5d5415bb16" containerID="6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64" exitCode=0
Sep 10 23:35:47.669462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.669304    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"
Sep 10 23:35:47.669462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.669336    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg" event={"ID":"110f6dd0-4140-4c6c-9653-9b5d5415bb16","Type":"ContainerDied","Data":"6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64"}
Sep 10 23:35:47.669462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.669391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg" event={"ID":"110f6dd0-4140-4c6c-9653-9b5d5415bb16","Type":"ContainerDied","Data":"2d5fcbc9e45c78446fec1234c7368167c629ecf96e28d398f14741a6c5605375"}
Sep 10 23:35:47.669462 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.669419    2577 scope.go:117] "RemoveContainer" containerID="6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64"
Sep 10 23:35:47.678293 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.678275    2577 scope.go:117] "RemoveContainer" containerID="6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64"
Sep 10 23:35:47.678532 ip-10-0-138-150 kubenswrapper[2577]: E0910 23:35:47.678517    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64\": container with ID starting with 6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64 not found: ID does not exist" containerID="6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64"
Sep 10 23:35:47.678581 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.678540    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64"} err="failed to get container status \"6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64\": rpc error: code = NotFound desc = could not find container \"6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64\": container with ID starting with 6f530fcadd3a111742772051751a34acb94020a95748acb342bcf5ebfeddbd64 not found: ID does not exist"
Sep 10 23:35:47.696312 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.696280    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"]
Sep 10 23:35:47.699983 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:47.699961    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-7f7f8945f9-45hfg"]
Sep 10 23:35:48.214767 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:35:48.214737    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110f6dd0-4140-4c6c-9653-9b5d5415bb16" path="/var/lib/kubelet/pods/110f6dd0-4140-4c6c-9653-9b5d5415bb16/volumes"
Sep 10 23:36:16.748586 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:16.748549    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5mt9l_50c25cd5-8dc4-4321-86e9-9d2a4a07bef5/manager/0.log"
Sep 10 23:36:16.996066 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:16.996033    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cf685499f-9dnbp_85148d71-b42f-42c9-9bcb-b69558766d02/manager/0.log"
Sep 10 23:36:17.105935 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:17.105861    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qtwzt_dd2fa529-5ccd-4881-b548-484513d752e5/server/0.log"
Sep 10 23:36:17.339315 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:17.339283    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfbc5c4d-96z5n_8349091b-3589-443f-961f-23d0a6155a5b/copy-manifests/0.log"
Sep 10 23:36:17.346008 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:17.345986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfbc5c4d-96z5n_8349091b-3589-443f-961f-23d0a6155a5b/manager/0.log"
Sep 10 23:36:18.026242 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:18.026213    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-559477f47-p29cf_b0fd6cba-336c-4cf8-8e98-8aae6e907fc3/maas-api/0.log"
Sep 10 23:36:18.346087 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:18.346012    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-c7xml_a351ffe3-6f4e-43bd-9563-78fde50c0490/postgres/0.log"
Sep 10 23:36:19.331630 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:19.331600    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-tvcjg_17823acd-5e64-4b9e-bbb8-0f028352b7f7/manager/0.log"
Sep 10 23:36:19.797319 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:19.797290    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-m76rp_17aacf39-4d0f-4402-99b9-63d60bb6422f/manager/0.log"
Sep 10 23:36:20.269270 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:20.269243    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jv9xt_5c8ba649-824d-442b-ad3a-2113c060dca6/discovery/0.log"
Sep 10 23:36:20.380121 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:20.380092    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9b555765-f2852_3a6f77ac-e13d-4a61-8f72-32128f0330d2/kube-auth-proxy/0.log"
Sep 10 23:36:20.737293 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:20.737261    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qxh2c_ca8081b0-eff4-4497-8af2-eaaf215d27a1/payload-pre-processing/0.log"
Sep 10 23:36:21.457122 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:21.457094    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-nzffn_6ba46953-bba7-4684-885e-d1f21261e9bb/main/0.log"
Sep 10 23:36:21.569651 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:21.569619    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-rjjgm_982bf7c2-483b-4c59-9f2a-9c4a0ca49773/main/0.log"
Sep 10 23:36:21.806932 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:21.806861    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fk5pz8_2bbbadb5-1335-448c-ae13-7798c53081e0/main/0.log"
Sep 10 23:36:26.934170 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:26.934132    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:27.034177 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:27.034143    2577 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log" limit=1024
Sep 10 23:36:30.269159 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:30.269131    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zd4wd_35acf209-c21a-4958-a3e9-0e951197405b/global-pull-secret-syncer/0.log"
Sep 10 23:36:30.318268 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:30.318240    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-88mdc_8d134860-6052-465b-92cd-ed4d065db2eb/konnectivity-agent/0.log"
Sep 10 23:36:30.467156 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:30.467129    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-150.ec2.internal_5bbd969cc419420da6c0edfe564045bc/haproxy/0.log"
Sep 10 23:36:35.033987 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:35.033954    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-tvcjg_17823acd-5e64-4b9e-bbb8-0f028352b7f7/manager/0.log"
Sep 10 23:36:35.215016 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:35.214955    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-m76rp_17aacf39-4d0f-4402-99b9-63d60bb6422f/manager/0.log"
Sep 10 23:36:37.349145 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:37.349104    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmmlc_f9b8494b-9f4b-41fd-b201-dd6aed67f685/node-exporter/0.log"
Sep 10 23:36:37.370385 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:37.370362    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmmlc_f9b8494b-9f4b-41fd-b201-dd6aed67f685/kube-rbac-proxy/0.log"
Sep 10 23:36:37.397053 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:37.397032    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-cmmlc_f9b8494b-9f4b-41fd-b201-dd6aed67f685/init-textfile/0.log"
Sep 10 23:36:39.621281 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:39.621253    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-5dcbc7df6b-zrj6s_9f799298-0931-4049-a366-034434be0df2/networking-console-plugin/0.log"
Sep 10 23:36:40.629956 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.629927    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"]
Sep 10 23:36:40.630309 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.630255    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110f6dd0-4140-4c6c-9653-9b5d5415bb16" containerName="maas-api"
Sep 10 23:36:40.630309 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.630265    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="110f6dd0-4140-4c6c-9653-9b5d5415bb16" containerName="maas-api"
Sep 10 23:36:40.630380 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.630329    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="110f6dd0-4140-4c6c-9653-9b5d5415bb16" containerName="maas-api"
Sep 10 23:36:40.633049 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.633027    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.635020 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.635000    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r674h\"/\"openshift-service-ca.crt\""
Sep 10 23:36:40.635145 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.635021    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-r674h\"/\"default-dockercfg-2cv5k\""
Sep 10 23:36:40.635706 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.635687    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-r674h\"/\"kube-root-ca.crt\""
Sep 10 23:36:40.642892 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.642873    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"]
Sep 10 23:36:40.693898 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.693871    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-sys\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.694078 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.693900    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft7js\" (UniqueName: \"kubernetes.io/projected/455de51e-10cd-41f3-99c2-801a79119b11-kube-api-access-ft7js\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.694078 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.694007    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-lib-modules\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.694078 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.694039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-proc\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.694078 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.694054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-podres\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795172 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795146    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-lib-modules\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795184    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-proc\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795202    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-podres\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795260    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-sys\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795450 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft7js\" (UniqueName: \"kubernetes.io/projected/455de51e-10cd-41f3-99c2-801a79119b11-kube-api-access-ft7js\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795450 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-lib-modules\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795450 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795347    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-sys\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795450 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-podres\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.795450 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.795364    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/455de51e-10cd-41f3-99c2-801a79119b11-proc\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.803421 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.803398    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft7js\" (UniqueName: \"kubernetes.io/projected/455de51e-10cd-41f3-99c2-801a79119b11-kube-api-access-ft7js\") pod \"perf-node-gather-daemonset-6kqj4\" (UID: \"455de51e-10cd-41f3-99c2-801a79119b11\") " pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:40.944097 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:40.944072    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:41.219129 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:41.215903    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"]
Sep 10 23:36:41.222317 ip-10-0-138-150 kubenswrapper[2577]: W0910 23:36:41.222290    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod455de51e_10cd_41f3_99c2_801a79119b11.slice/crio-d627800b30986ffa491afda0416faa0a7abb682fa9f666dac91c302e5b97b533 WatchSource:0}: Error finding container d627800b30986ffa491afda0416faa0a7abb682fa9f666dac91c302e5b97b533: Status 404 returned error can't find the container with id d627800b30986ffa491afda0416faa0a7abb682fa9f666dac91c302e5b97b533
Sep 10 23:36:41.917042 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:41.917003    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4" event={"ID":"455de51e-10cd-41f3-99c2-801a79119b11","Type":"ContainerStarted","Data":"ceef15037b95570417004373ab3370fb6e45373fc627ada464d091222dea9088"}
Sep 10 23:36:41.917407 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:41.917063    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4" event={"ID":"455de51e-10cd-41f3-99c2-801a79119b11","Type":"ContainerStarted","Data":"d627800b30986ffa491afda0416faa0a7abb682fa9f666dac91c302e5b97b533"}
Sep 10 23:36:41.917407 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:41.917121    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:41.933903 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:41.933857    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4" podStartSLOduration=1.933845361 podStartE2EDuration="1.933845361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 23:36:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 23:36:41.932889801 +0000 UTC m=+2318.368944598" watchObservedRunningTime="2026-09-10 23:36:41.933845361 +0000 UTC m=+2318.369900183"
Sep 10 23:36:42.782296 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:42.782267    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqndg_e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab/dns/0.log"
Sep 10 23:36:42.805389 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:42.805359    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sqndg_e3b6ccc7-12b0-4b07-8ea8-32e4295e19ab/kube-rbac-proxy/0.log"
Sep 10 23:36:42.936887 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:42.936858    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jh4r7_5cebf87f-515b-4607-8e4e-1d53ed85cfda/dns-node-resolver/0.log"
Sep 10 23:36:43.582069 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:43.582041    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9hrxg_ddf1e8ab-c897-4333-8140-78d9d49493bc/node-ca/0.log"
Sep 10 23:36:44.780303 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:44.780270    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jv9xt_5c8ba649-824d-442b-ad3a-2113c060dca6/discovery/0.log"
Sep 10 23:36:44.811841 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:44.811812    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9b555765-f2852_3a6f77ac-e13d-4a61-8f72-32128f0330d2/kube-auth-proxy/0.log"
Sep 10 23:36:44.931794 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:44.931747    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qxh2c_ca8081b0-eff4-4497-8af2-eaaf215d27a1/payload-pre-processing/0.log"
Sep 10 23:36:45.932685 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:45.932656    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vlhs2_3f5cfece-be37-4c58-80b8-d099ea39e781/serve-healthcheck-canary/0.log"
Sep 10 23:36:46.676378 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:46.676350    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxjdz_0d1191d1-a838-42a3-a01b-045e73652543/kube-rbac-proxy/0.log"
Sep 10 23:36:46.698912 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:46.698890    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxjdz_0d1191d1-a838-42a3-a01b-045e73652543/exporter/0.log"
Sep 10 23:36:46.721791 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:46.721759    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hxjdz_0d1191d1-a838-42a3-a01b-045e73652543/extractor/0.log"
Sep 10 23:36:47.931344 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:47.931316    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-r674h/perf-node-gather-daemonset-6kqj4"
Sep 10 23:36:48.862428 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:48.862403    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5mt9l_50c25cd5-8dc4-4321-86e9-9d2a4a07bef5/manager/0.log"
Sep 10 23:36:49.142136 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:49.142053    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-cf685499f-9dnbp_85148d71-b42f-42c9-9bcb-b69558766d02/manager/0.log"
Sep 10 23:36:49.172340 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:49.172309    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-qtwzt_dd2fa529-5ccd-4881-b548-484513d752e5/server/0.log"
Sep 10 23:36:49.316009 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:49.315986    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfbc5c4d-96z5n_8349091b-3589-443f-961f-23d0a6155a5b/manager/0.log"
Sep 10 23:36:49.344924 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:49.344906    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-dbfbc5c4d-96z5n_8349091b-3589-443f-961f-23d0a6155a5b/copy-manifests/0.log"
Sep 10 23:36:51.213125 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:51.213088    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jv9xt_5c8ba649-824d-442b-ad3a-2113c060dca6/discovery/0.log"
Sep 10 23:36:51.226868 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:51.226847    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9b555765-f2852_3a6f77ac-e13d-4a61-8f72-32128f0330d2/kube-auth-proxy/0.log"
Sep 10 23:36:51.308527 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:51.308492    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qxh2c_ca8081b0-eff4-4497-8af2-eaaf215d27a1/payload-pre-processing/0.log"
Sep 10 23:36:53.081088 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:53.081052    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-jv9xt_5c8ba649-824d-442b-ad3a-2113c060dca6/discovery/0.log"
Sep 10 23:36:53.095172 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:53.095148    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-7d9b555765-f2852_3a6f77ac-e13d-4a61-8f72-32128f0330d2/kube-auth-proxy/0.log"
Sep 10 23:36:53.169542 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:53.169510    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-qxh2c_ca8081b0-eff4-4497-8af2-eaaf215d27a1/payload-pre-processing/0.log"
Sep 10 23:36:56.752689 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:56.752650    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9nj9m_9676f20c-a73c-4e02-97dd-9c12b6792649/kube-storage-version-migrator-operator/1.log"
Sep 10 23:36:56.753471 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:56.753450    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6c8f84f897-9nj9m_9676f20c-a73c-4e02-97dd-9c12b6792649/kube-storage-version-migrator-operator/0.log"
Sep 10 23:36:58.179025 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:58.178998    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6gr9_8a03f169-6f4e-4c71-aedd-aa79c1e9e859/kube-multus-additional-cni-plugins/0.log"
Sep 10 23:36:58.204615 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:58.204589    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6gr9_8a03f169-6f4e-4c71-aedd-aa79c1e9e859/egress-router-binary-copy/0.log"
Sep 10 23:36:58.228044 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:58.228022    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6gr9_8a03f169-6f4e-4c71-aedd-aa79c1e9e859/cni-plugins/0.log"
Sep 10 23:36:58.251125 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:58.251100    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6gr9_8a03f169-6f4e-4c71-aedd-aa79c1e9e859/bond-cni-plugin/0.log"
Sep 10 23:36:58.274491 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:58.274461    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6gr9_8a03f169-6f4e-4c71-aedd-aa79c1e9e859/routeoverride-cni/0.log"
Sep 10 23:36:58.298127 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:58.298097    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6gr9_8a03f169-6f4e-4c71-aedd-aa79c1e9e859/whereabouts-cni-bincopy/0.log"
Sep 10 23:36:58.325200 ip-10-0-138-150 kubenswrapper[2577]: I0910 23:36:58.325173    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-t6gr9_8a03f169-6f4e-4c71-aedd-aa79c1e9e859/whereabouts-cni/0.log"