Sep 10 15:32:36.265056 ip-10-0-139-11 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 15:32:36.685076 ip-10-0-139-11 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:32:36.685076 ip-10-0-139-11 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 15:32:36.685076 ip-10-0-139-11 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:32:36.685076 ip-10-0-139-11 kubenswrapper[2577]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 15:32:36.685076 ip-10-0-139-11 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:32:36.685902 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.685810    2577 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 15:32:36.688912 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688897    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:32:36.688912 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688911    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688915    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688919    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688922    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688925    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688928    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688931    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688933    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688936    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688939    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688946    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688949    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688952    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688954    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688957    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688966    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688969    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688972    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688974    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688977    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:32:36.688984 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688979    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688982    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688985    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688987    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688990    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688993    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688996    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.688998    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689001    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689003    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689006    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689008    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689011    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689013    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689016    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689018    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689021    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689023    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689025    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689028    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:32:36.689462 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689031    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689034    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689036    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689039    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689041    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689043    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689046    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689048    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689050    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689053    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689055    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689057    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689060    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689062    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689065    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689070    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689074    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689077    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689080    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:32:36.689976 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689083    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689086    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689089    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689091    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689094    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689097    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689099    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689102    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689104    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689107    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689109    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689112    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689115    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689118    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689121    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689124    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689126    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689129    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689132    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689136    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:32:36.690461 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689139    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689142    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689144    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689147    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689149    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689152    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689154    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689157    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689160    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689162    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689166    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689169    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689171    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689174    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689178    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689181    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689183    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689186    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689189    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689905    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:32:36.691220 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689912    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689915    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689919    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689922    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689924    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689927    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689930    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689932    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689935    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689938    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689941    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689945    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689947    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689950    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689952    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689955    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689958    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689960    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689963    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689966    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:32:36.691919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689969    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689971    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689974    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689977    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689979    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689982    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689985    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689987    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689990    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689993    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689995    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.689997    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690003    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690006    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690010    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690012    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690015    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690018    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690020    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:32:36.692412 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690024    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690026    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690030    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690033    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690035    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690038    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690041    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690043    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690046    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690048    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690050    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690053    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690056    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690058    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690061    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690063    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690066    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690068    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690071    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690073    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:32:36.692919 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690076    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690078    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690080    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690083    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690086    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690088    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690091    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690093    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690095    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690098    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690102    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690105    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690107    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690110    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690112    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690115    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690118    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690121    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690123    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690126    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:32:36.693574 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690128    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690131    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690133    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690135    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690137    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690140    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690142    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690145    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690149    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690153    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690156    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690159    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690161    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690164    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690166    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690169    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690172    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690174    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690176    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:32:36.694476 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.690699    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693287    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693307    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693313    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693339    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693448    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693456    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693460    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693465    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693470    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693474    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693478    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693483    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693487    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693491    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693495    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693501    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693506    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693510    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693514    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:32:36.695313 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693518    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693523    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693527    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693531    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693536    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693540    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693545    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693550    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693555    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693559    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693563    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693567    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693571    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693576    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693580    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693598    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693606    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693619    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693625    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:32:36.695926 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693630    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693635    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693640    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693644    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693649    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693653    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693659    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693665    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693670    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693674    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693678    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693686    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693690    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693694    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693699    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693702    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693707    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693711    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693716    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:32:36.696490 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693720    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693724    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693729    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693735    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693740    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693745    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693770    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693775    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693780    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693785    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693789    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693794    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693798    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693802    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693806    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693810    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693814    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693818    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693822    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693828    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:32:36.696998 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693832    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693839    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693843    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693847    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693851    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693855    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693859    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693863    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693867    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693871    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693876    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693880    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693884    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693888    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693892    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693897    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693901    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693905    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693909    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693914    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:32:36.697506 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.693918    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694027    2577 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694079    2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694089    2577 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694096    2577 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694103    2577 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694108    2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694115    2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694125    2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694130    2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694135    2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694140    2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694145    2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694150    2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694155    2577 flags.go:64] FLAG: --cgroup-root=""
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694160    2577 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694164    2577 flags.go:64] FLAG: --client-ca-file=""
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694169    2577 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694173    2577 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694181    2577 flags.go:64] FLAG: --cluster-domain=""
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694185    2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694190    2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694196    2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694201    2577 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 15:32:36.698203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694207    2577 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694212    2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694217    2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694222    2577 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694227    2577 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694232    2577 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694236    2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694241    2577 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694247    2577 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694253    2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694259    2577 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694264    2577 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694268    2577 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694273    2577 flags.go:64] FLAG: --enable-server="true"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694278    2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694284    2577 flags.go:64] FLAG: --event-burst="100"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694289    2577 flags.go:64] FLAG: --event-qps="50"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694294    2577 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694299    2577 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694303    2577 flags.go:64] FLAG: --eviction-hard=""
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694309    2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694314    2577 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694319    2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694323    2577 flags.go:64] FLAG: --eviction-soft=""
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694328    2577 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 15:32:36.698847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694333    2577 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694337    2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694342    2577 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694347    2577 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694351    2577 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694356    2577 flags.go:64] FLAG: --feature-gates=""
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694363    2577 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694367    2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694372    2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694377    2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694382    2577 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694387    2577 flags.go:64] FLAG: --help="false"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694391    2577 flags.go:64] FLAG: --hostname-override="ip-10-0-139-11.ec2.internal"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694397    2577 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694402    2577 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694408    2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694414    2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694420    2577 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694424    2577 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694429    2577 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694434    2577 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694438    2577 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694443    2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694448    2577 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 15:32:36.699595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694453    2577 flags.go:64] FLAG: --kube-reserved=""
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694457    2577 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694462    2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694467    2577 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694471    2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694476    2577 flags.go:64] FLAG: --lock-file=""
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694480    2577 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694484    2577 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694489    2577 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694497    2577 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694502    2577 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694507    2577 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694512    2577 flags.go:64] FLAG: --logging-format="text"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694517    2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694523    2577 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694528    2577 flags.go:64] FLAG: --manifest-url=""
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694532    2577 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694539    2577 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694543    2577 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694550    2577 flags.go:64] FLAG: --max-pods="110"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694555    2577 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694560    2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694565    2577 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694570    2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694574    2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 15:32:36.700241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694579    2577 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694583    2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694594    2577 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694599    2577 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694604    2577 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694608    2577 flags.go:64] FLAG: --pod-cidr=""
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694613    2577 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694621    2577 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694625    2577 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694630    2577 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694634    2577 flags.go:64] FLAG: --port="10250"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694639    2577 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694645    2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-02527cce7ac49f1ab"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694650    2577 flags.go:64] FLAG: --qos-reserved=""
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694655    2577 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694660    2577 flags.go:64] FLAG: --register-node="true"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694664    2577 flags.go:64] FLAG: --register-with-taints=""
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694670    2577 flags.go:64] FLAG: --registry-burst="10"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694674    2577 flags.go:64] FLAG: --registry-qps="5"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694681    2577 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694686    2577 flags.go:64] FLAG: --reserved-memory=""
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694691    2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694696    2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694701    2577 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694705    2577 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 15:32:36.700877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694710    2577 flags.go:64] FLAG: --runonce="false"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694714    2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694719    2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694724    2577 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694728    2577 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694737    2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694742    2577 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694765    2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694770    2577 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694775    2577 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694780    2577 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694785    2577 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694789    2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694795    2577 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694799    2577 flags.go:64] FLAG: --system-cgroups=""
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694804    2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694813    2577 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694818    2577 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694823    2577 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694839    2577 flags.go:64] FLAG: --tls-min-version=""
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694844    2577 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694848    2577 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694853    2577 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694858    2577 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694862    2577 flags.go:64] FLAG: --v="2"
Sep 10 15:32:36.701485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694876    2577 flags.go:64] FLAG: --version="false"
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694882    2577 flags.go:64] FLAG: --vmodule=""
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694892    2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.694897    2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695084    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695092    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695096    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695100    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695104    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695108    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695113    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695117    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695123    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695127    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695131    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695135    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695139    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695143    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695147    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695152    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:32:36.702156 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695156    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695161    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695165    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695169    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695173    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695177    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695181    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695185    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695192    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695198    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695203    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695207    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695211    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695215    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695219    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695224    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695229    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695235    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695240    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695245    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:32:36.702742 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695249    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695253    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695257    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695261    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695267    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695271    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695275    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695279    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695283    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695287    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695291    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695295    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695300    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695306    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695310    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695313    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695320    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695325    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695330    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:32:36.703301 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695334    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695339    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695343    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695347    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695352    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695356    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695360    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695365    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695369    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695375    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695379    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695384    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695388    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695391    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695396    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695400    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695404    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695410    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695415    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695419    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:32:36.703778 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695423    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695426    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695430    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695434    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695438    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695443    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695448    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695452    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695456    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695460    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695464    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695468    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695472    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695476    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695481    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695485    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695489    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695493    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695497    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695502    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:32:36.704269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695506    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695510    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695516    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.695520    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.696236    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.703857    2577 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.703874    2577 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.703888    2577 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.703894    2577 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703950    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703955    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703958    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703961    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703963    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703966    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:32:36.704852 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703969    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703971    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703974    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703976    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703979    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703981    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703984    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703986    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703989    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703992    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703994    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703997    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.703999    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704002    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704004    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704006    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704009    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704011    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704014    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:32:36.705230 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704018    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704022    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704024    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704027    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704031    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704034    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704037    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704039    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704044    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704047    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704050    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704053    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704056    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704059    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704061    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704064    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704066    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704069    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704071    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:32:36.705694 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704074    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704076    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704078    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704081    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704083    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704086    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704088    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704091    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704093    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704095    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704098    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704101    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704103    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704106    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704108    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704110    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704113    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704116    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704119    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704122    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:32:36.706185 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704124    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704127    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704129    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704131    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704134    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704136    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704139    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704141    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704144    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704147    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704149    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704152    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704154    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704157    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704159    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704162    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704165    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704167    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704170    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704172    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:32:36.706675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704175    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704177    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704179    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704182    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704184    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704188    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704191    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704194    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704197    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704200    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704203    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704206    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704208    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704211    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704213    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:32:36.707223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.704218    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704332    2577 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704337    2577 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704339    2577 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704343    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704345    2577 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704349    2577 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704351    2577 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704354    2577 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704357    2577 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704360    2577 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704363    2577 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704365    2577 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704368    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704371    2577 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704373    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704376    2577 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704379    2577 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704381    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704384    2577 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:32:36.707598 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704386    2577 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704389    2577 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704391    2577 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704394    2577 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704396    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704399    2577 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704401    2577 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704404    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704407    2577 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704410    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704412    2577 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704415    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704417    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704420    2577 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704423    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704425    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704428    2577 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704430    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704432    2577 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704435    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:32:36.708102 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704437    2577 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704439    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704442    2577 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704444    2577 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704446    2577 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704450    2577 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704454    2577 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704456    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704459    2577 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704462    2577 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704465    2577 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704467    2577 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704470    2577 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704472    2577 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704475    2577 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704477    2577 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704480    2577 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704482    2577 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704485    2577 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:32:36.708616 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704487    2577 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704489    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704492    2577 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704499    2577 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704502    2577 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704504    2577 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704507    2577 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704509    2577 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704512    2577 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704514    2577 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704517    2577 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704519    2577 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704522    2577 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704524    2577 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704527    2577 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704529    2577 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704532    2577 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704534    2577 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704538    2577 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:32:36.709275 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704541    2577 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704544    2577 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704547    2577 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704550    2577 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704554    2577 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704557    2577 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704560    2577 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704563    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704565    2577 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704568    2577 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704570    2577 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704573    2577 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704575    2577 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704578    2577 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704581    2577 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704584    2577 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704586    2577 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704589    2577 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704592    2577 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:32:36.709802 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704594    2577 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704596    2577 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:36.704599    2577 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.704604    2577 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.704721    2577 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.706956    2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.708598    2577 server.go:1019] "Starting client certificate rotation"
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.708708    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:32:36.710248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.708771    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:32:36.713569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.713550    2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:32:36.714698 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.714676    2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:32:36.724998 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.724980    2577 log.go:25] "Validated CRI v1 runtime API"
Sep 10 15:32:36.727957 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.727940    2577 log.go:25] "Validated CRI v1 image API"
Sep 10 15:32:36.728876 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.728858    2577 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 15:32:36.732685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.732659    2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 d63333f8-37fc-4c7a-9fe4-8857fe86e141:/dev/nvme0n1p4 e7c9d359-6c13-4809-acb7-f1c080fbfd33:/dev/nvme0n1p3]
Sep 10 15:32:36.732776 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.732684    2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 15:32:36.738955 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.738844    2577 manager.go:216] Machine: {Timestamp:2026-09-10 15:32:36.737602675 +0000 UTC m=+0.351729571 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100419 MemoryCapacity:32812146688 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec284b9869cec9519fe6a61f23f18a94 SystemUUID:ec284b98-69ce-c951-9fe6-a61f23f18a94 BootID:6309a4db-4b8d-4791-90c8-4f031a6be59f Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406073344 Type:vfs Inodes:4005389 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562430976 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406073344 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e0:0d:d6:2b:45 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e0:0d:d6:2b:45 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:1a:69:07:e4:e5:38 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812146688 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 15:32:36.738955 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.738950    2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 15:32:36.739068 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.739030    2577 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 15:32:36.741420 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.741405    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:32:36.742660 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.742635    2577 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 15:32:36.742816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.742663    2577 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-11.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 15:32:36.743590 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.743571    2577 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 15:32:36.743590 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.743585    2577 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 15:32:36.743590 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.743594    2577 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:32:36.744860 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.744849    2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:32:36.746828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.746818    2577 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 15:32:36.750762 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.750735    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:32:36.750870 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.750860    2577 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 15:32:36.750902 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.750890    2577 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 15:32:36.750902 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.750900    2577 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 15:32:36.750957 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.750911    2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 15:32:36.750957 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.750919    2577 kubelet.go:398] "Adding apiserver pod source"
Sep 10 15:32:36.750957 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.750928    2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 15:32:36.751310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.751299    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:32:36.751359 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.751318    2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:32:36.754180 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.754158    2577 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 15:32:36.757034 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757020    2577 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 15:32:36.757099 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757050    2577 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 15:32:36.757214 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757204    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 15:32:36.757250 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757220    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 15:32:36.757250 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757226    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 15:32:36.757250 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757233    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 15:32:36.757250 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757244    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 15:32:36.757363 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757253    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 15:32:36.757947 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757937    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 15:32:36.757947 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757947    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 15:32:36.758008 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757955    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 15:32:36.758008 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757962    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 15:32:36.758008 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757976    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 15:32:36.758008 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.757984    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 15:32:36.759466 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.759454    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 15:32:36.759504 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.759470    2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 15:32:36.763740 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.763724    2577 server.go:1268] "Started kubelet"
Sep 10 15:32:36.763838 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.763792    2577 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 15:32:36.764410 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.764331    2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 15:32:36.764410 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.764397    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 15:32:36.764554 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.764397    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 15:32:36.764554 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.764427    2577 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 15:32:36.764554 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.764501    2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-11.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 15:32:36.765425 ip-10-0-139-11 systemd[1]: Started Kubernetes Kubelet.
Sep 10 15:32:36.765791 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.765774    2577 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 15:32:36.766737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.766723    2577 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 15:32:36.769271 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.769221    2577 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 15:32:36.771348 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.771331    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nw9c5"
Sep 10 15:32:36.771513 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.770624    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-11.ec2.internal.18d3ff756ac3c6bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-11.ec2.internal,UID:ip-10-0-139-11.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-11.ec2.internal,},FirstTimestamp:2026-09-10 15:32:36.763690683 +0000 UTC m=+0.377817579,LastTimestamp:2026-09-10 15:32:36.763690683 +0000 UTC m=+0.377817579,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-11.ec2.internal,}"
Sep 10 15:32:36.773844 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.773822    2577 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 15:32:36.773938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.773833    2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 15:32:36.773992 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.773954    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:36.774733 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.774715    2577 factory.go:55] Registering systemd factory
Sep 10 15:32:36.774833 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.774746    2577 factory.go:223] Registration of the systemd container factory successfully
Sep 10 15:32:36.775569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775383    2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 15:32:36.775655 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775384    2577 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 15:32:36.775655 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775583    2577 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 15:32:36.775655 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775507    2577 factory.go:153] Registering CRI-O factory
Sep 10 15:32:36.775655 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775618    2577 factory.go:223] Registration of the crio container factory successfully
Sep 10 15:32:36.775868 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775674    2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 15:32:36.775868 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775686    2577 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 15:32:36.775868 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775694    2577 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 15:32:36.775868 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775695    2577 factory.go:103] Registering Raw factory
Sep 10 15:32:36.775868 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.775723    2577 manager.go:1196] Started watching for new ooms in manager
Sep 10 15:32:36.776121 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.776107    2577 manager.go:349] Starting recovery of all containers
Sep 10 15:32:36.777727 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.777701    2577 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 15:32:36.777968 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.777930    2577 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-11.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 15:32:36.782711 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.782687    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nw9c5"
Sep 10 15:32:36.784892 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.784875    2577 manager.go:354] Recovery completed
Sep 10 15:32:36.786220 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.786195    2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": readdirent /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 15:32:36.789014 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.789002    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:32:36.790804 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.790790    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:32:36.790881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.790825    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:32:36.790881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.790841    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:32:36.791420 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.791406    2577 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 15:32:36.791420 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.791418    2577 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 15:32:36.791521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.791439    2577 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:32:36.791521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.791419    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:32:36.792512 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.792494    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:32:36.792580 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.792523    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:32:36.792580 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.792535    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:32:36.792846 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.792835    2577 policy_none.go:49] "None policy: Start"
Sep 10 15:32:36.792884 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.792850    2577 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 15:32:36.792884 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.792860    2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 15:32:36.793173 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.793115    2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-11.ec2.internal.18d3ff756c617f86  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-11.ec2.internal,UID:ip-10-0-139-11.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-11.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-11.ec2.internal,},FirstTimestamp:2026-09-10 15:32:36.790804358 +0000 UTC m=+0.404931255,LastTimestamp:2026-09-10 15:32:36.790804358 +0000 UTC m=+0.404931255,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-11.ec2.internal,}"
Sep 10 15:32:36.794440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.794428    2577 policy_none.go:47] "Start"
Sep 10 15:32:36.836334 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.836317    2577 manager.go:337] "Starting Device Plugin manager"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.836361    2577 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.836374    2577 server.go:87] "Starting device plugin registration server"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.836683    2577 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.836694    2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.836806    2577 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.836890    2577 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.836899    2577 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.837337    2577 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 15:32:36.845701 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.837375    2577 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:36.899919 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.899886    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 15:32:36.901235 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.901217    2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 15:32:36.901308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.901280    2577 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 15:32:36.901355 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.901316    2577 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 15:32:36.901387 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.901357    2577 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 15:32:36.905490 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.905469    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:32:36.937474 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.937373    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:32:36.938395 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.938379    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:32:36.938472 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.938409    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:32:36.938472 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.938419    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:32:36.938472 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.938441    2577 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-11.ec2.internal"
Sep 10 15:32:36.944918 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:36.944893    2577 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-11.ec2.internal"
Sep 10 15:32:36.945030 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.944920    2577 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-11.ec2.internal\": node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:36.961405 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:36.961380    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.001685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.001652    2577 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"]
Sep 10 15:32:37.001819 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.001727    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:32:37.003362 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.003346    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:32:37.003449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.003380    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:32:37.003449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.003394    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:32:37.004695 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.004678    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:32:37.004811 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.004797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.004864 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.004835    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:32:37.006419 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.005922    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:32:37.006419 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.005955    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:32:37.006419 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.005969    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:32:37.006590 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.006471    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:32:37.006590 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.006510    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:32:37.006590 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.006522    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:32:37.007252 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.007235    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.007336 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.007270    2577 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:32:37.007996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.007982    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:32:37.008067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.008006    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:32:37.008067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.008017    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:32:37.023297 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.023277    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-11.ec2.internal\" not found" node="ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.026973 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.026958    2577 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-11.ec2.internal\" not found" node="ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.061799 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.061775    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.077142 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.077117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b183149671d7e63b663b38bcacd05dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-11.ec2.internal\" (UID: \"b183149671d7e63b663b38bcacd05dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.077226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.077142    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c65e95aed64261e20e4361dcc2a37963-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal\" (UID: \"c65e95aed64261e20e4361dcc2a37963\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.077226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.077163    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c65e95aed64261e20e4361dcc2a37963-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal\" (UID: \"c65e95aed64261e20e4361dcc2a37963\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.162644 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.162602    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.178065 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.178041    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b183149671d7e63b663b38bcacd05dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-11.ec2.internal\" (UID: \"b183149671d7e63b663b38bcacd05dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.178115 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.178071    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c65e95aed64261e20e4361dcc2a37963-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal\" (UID: \"c65e95aed64261e20e4361dcc2a37963\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.178115 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.178088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c65e95aed64261e20e4361dcc2a37963-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal\" (UID: \"c65e95aed64261e20e4361dcc2a37963\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.178176 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.178146    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/c65e95aed64261e20e4361dcc2a37963-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal\" (UID: \"c65e95aed64261e20e4361dcc2a37963\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.178176 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.178150    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b183149671d7e63b663b38bcacd05dd5-config\") pod \"kube-apiserver-proxy-ip-10-0-139-11.ec2.internal\" (UID: \"b183149671d7e63b663b38bcacd05dd5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.178238 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.178180    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c65e95aed64261e20e4361dcc2a37963-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal\" (UID: \"c65e95aed64261e20e4361dcc2a37963\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.263521 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.263448    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.327039 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.327007    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.329682 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.329663    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.364114 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.364088    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.464706 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.464676    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.565329 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.565255    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.665915 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.665881    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.707445 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.707419    2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 15:32:37.707930 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.707616    2577 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 15:32:37.766459 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.766433    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.785156 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.785122    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:27:37 +0000 UTC" deadline="2028-06-20 18:47:40.261191188 +0000 UTC"
Sep 10 15:32:37.785156 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.785153    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15579h15m2.476042232s"
Sep 10 15:32:37.867507 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:37.867439    2577 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-11.ec2.internal\" not found"
Sep 10 15:32:37.957258 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.957233    2577 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:32:37.974344 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.974320    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:37.989901 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.989877    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:32:37.991870 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:37.991853    2577 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal"
Sep 10 15:32:38.002181 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.002163    2577 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:32:38.021427 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:38.021391    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb183149671d7e63b663b38bcacd05dd5.slice/crio-d5d68373bb285fa09b07e467fe2e47719840110434df4713a4ebe32012bb0acc WatchSource:0}: Error finding container d5d68373bb285fa09b07e467fe2e47719840110434df4713a4ebe32012bb0acc: Status 404 returned error can't find the container with id d5d68373bb285fa09b07e467fe2e47719840110434df4713a4ebe32012bb0acc
Sep 10 15:32:38.021641 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:38.021621    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc65e95aed64261e20e4361dcc2a37963.slice/crio-71b5c7ec3bd07022fd96997dba584285dda09619eacee9172a5b02022d6ee39f WatchSource:0}: Error finding container 71b5c7ec3bd07022fd96997dba584285dda09619eacee9172a5b02022d6ee39f: Status 404 returned error can't find the container with id 71b5c7ec3bd07022fd96997dba584285dda09619eacee9172a5b02022d6ee39f
Sep 10 15:32:38.027428 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.027413    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:32:38.154310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.154215    2577 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:32:38.367217 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.367183    2577 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:32:38.752136 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.752061    2577 apiserver.go:52] "Watching apiserver"
Sep 10 15:32:38.758100 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.758071    2577 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 15:32:38.758636 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.758612    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/konnectivity-agent-5d7px","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp","openshift-dns/node-resolver-hm9kh","openshift-multus/multus-additional-cni-plugins-vzwtb","openshift-multus/multus-n52xc","openshift-multus/network-metrics-daemon-qpkv8","kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal","openshift-cluster-node-tuning-operator/tuned-fb8nl","openshift-image-registry/node-ca-jbtrv","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal","openshift-network-diagnostics/network-check-target-dx6np","openshift-network-operator/iptables-alerter-8sc54","openshift-ovn-kubernetes/ovnkube-node-dk4jq","kube-system/global-pull-secret-syncer-j5nr8"]
Sep 10 15:32:38.761803 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.761775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.765030 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.764975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.767452 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.767423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.768620 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.768596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.769687 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.769666    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.770232 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.770210    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zzx8r\""
Sep 10 15:32:38.770315 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.770263    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.770518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.770498    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-m4rc6\""
Sep 10 15:32:38.772087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.770716    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.772087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.770936    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.772087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.771071    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.772087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.771240    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 15:32:38.772087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.771464    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.772087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.771628    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.772087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.771893    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6f4kq\""
Sep 10 15:32:38.772488 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.772469    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.772704    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.772748    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.772986    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773009    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773320    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ktlhh\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773349    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773427    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773554    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.773708 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773584    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-mcqq5\""
Sep 10 15:32:38.774286 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.773910    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 15:32:38.774286 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.774129    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-p7v6p\""
Sep 10 15:32:38.774721 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.774698    2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 15:32:38.774897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.774873    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.775487 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.775468    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:38.775576 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.775560    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:38.776094 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.776074    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:38.776183 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.776147    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:38.777705 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.777688    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.778845 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.778828    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.780338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.780314    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.780434 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.780379    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:38.781994 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.781971    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 15:32:38.781994 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.781987    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.782348 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782328    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 15:32:38.782598 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782579    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-prxxk\""
Sep 10 15:32:38.782683 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782661    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 15:32:38.782772 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782755    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.782855 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782806    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 15:32:38.782935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782921    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 15:32:38.782999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782673    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 15:32:38.782999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.782974    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-rz49d\""
Sep 10 15:32:38.783240 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.783218    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ptkmb\""
Sep 10 15:32:38.783432 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.783414    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 15:32:38.783598 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.783434    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 15:32:38.783685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.783422    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 15:32:38.787696 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysconfig\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.787795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787714    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9397095-0672-4769-8a7b-d21de088e931-etc-tuned\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.787795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lw4bw\" (UniqueName: \"kubernetes.io/projected/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-kube-api-access-lw4bw\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:38.787941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.787941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787819    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.787941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787842    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9397095-0672-4769-8a7b-d21de088e931-tmp\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.787941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-multus-certs\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.787941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.787941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-cni-binary-copy\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787947    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-socket-dir-parent\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-lib-modules\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.787992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-cni-bin\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788012    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-kubernetes\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788034    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0726f3e-2027-4c55-8f1a-7d9c73e46994-tmp-dir\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffxrv\" (UniqueName: \"kubernetes.io/projected/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-kube-api-access-ffxrv\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788115    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-cni-multus\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rggcm\" (UniqueName: \"kubernetes.io/projected/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-kube-api-access-rggcm\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788146    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-device-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788160    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07790bc9-5df8-4bb9-8739-afe6fe8b9473-iptables-alerter-script\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788221    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-sys\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788243    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-etc-kubernetes\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788257    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-system-cni-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788277    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-conf-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.788360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-serviceca\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.788888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788795    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04cdccb3-c41a-423d-8a73-ffd42a5ccaa9-agent-certs\") pod \"konnectivity-agent-5d7px\" (UID: \"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9\") " pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:38.788888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07790bc9-5df8-4bb9-8739-afe6fe8b9473-host-slash\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.788888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-socket-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.789038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788905    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysctl-d\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788929    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-systemd\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788952    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-var-lib-kubelet\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.788976    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cnibin\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.789038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789018    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-hostroot\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.789277 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789047    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thmzq\" (UniqueName: \"kubernetes.io/projected/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-kube-api-access-thmzq\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.789277 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789091    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:38.789277 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789126    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysctl-conf\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789277 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789157    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.789277 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-host\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.789518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789296    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04cdccb3-c41a-423d-8a73-ffd42a5ccaa9-konnectivity-ca\") pod \"konnectivity-agent-5d7px\" (UID: \"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9\") " pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:38.789518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789322    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-sys-fs\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.789518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789366    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft9nf\" (UniqueName: \"kubernetes.io/projected/577e86c4-8d3c-448c-bc58-089749f835c5-kube-api-access-ft9nf\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.789518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0726f3e-2027-4c55-8f1a-7d9c73e46994-hosts-file\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.789518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789450    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-system-cni-dir\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.789518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-host\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-modprobe-d\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789523    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-run\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789550    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmcmk\" (UniqueName: \"kubernetes.io/projected/c0726f3e-2027-4c55-8f1a-7d9c73e46994-kube-api-access-gmcmk\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789592    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-k8s-cni-cncf-io\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789630    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-netns\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789655    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgffw\" (UniqueName: \"kubernetes.io/projected/07790bc9-5df8-4bb9-8739-afe6fe8b9473-kube-api-access-zgffw\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789676    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8xkg\" (UniqueName: \"kubernetes.io/projected/d9397095-0672-4769-8a7b-d21de088e931-kube-api-access-s8xkg\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789708    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-kubelet\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789740    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789787    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-cni-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.789875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789836    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-cnibin\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.790420 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789885    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:38.790420 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789916    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-daemon-config\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.790420 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.789940    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-registration-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.791926 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.791525    2577 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:32:38.811961 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.811936    2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zkdpp"
Sep 10 15:32:38.820381 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.820361    2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zkdpp"
Sep 10 15:32:38.877038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.877003    2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 15:32:38.890343 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890304    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-etc-kubernetes\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.890493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890355    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-kubelet\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.890493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890381    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-systemd\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.890493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890407    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-var-lib-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.890493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890434    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-system-cni-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.890493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-conf-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.890493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890485    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-serviceca\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-env-overrides\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890559    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d4644fb-55db-4320-9d71-ca7b885427f5-dbus\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890622    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04cdccb3-c41a-423d-8a73-ffd42a5ccaa9-agent-certs\") pod \"konnectivity-agent-5d7px\" (UID: \"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9\") " pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890649    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07790bc9-5df8-4bb9-8739-afe6fe8b9473-host-slash\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890677    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-socket-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890702    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysctl-d\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890726    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-systemd\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890752    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-var-lib-kubelet\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.890816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890791    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cnibin\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890835    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-hostroot\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thmzq\" (UniqueName: \"kubernetes.io/projected/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-kube-api-access-thmzq\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890886    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890902    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysctl-conf\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890918    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-host\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890970    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04cdccb3-c41a-423d-8a73-ffd42a5ccaa9-konnectivity-ca\") pod \"konnectivity-agent-5d7px\" (UID: \"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9\") " pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.890995    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-sys-fs\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891036    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft9nf\" (UniqueName: \"kubernetes.io/projected/577e86c4-8d3c-448c-bc58-089749f835c5-kube-api-access-ft9nf\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovn-node-metrics-cert\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0726f3e-2027-4c55-8f1a-7d9c73e46994-hosts-file\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891115    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-system-cni-dir\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891139    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-run-netns\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891164    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-etc-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891191    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovnkube-config\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.891236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891216    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-host\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891250    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-modprobe-d\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-run\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891316    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmcmk\" (UniqueName: \"kubernetes.io/projected/c0726f3e-2027-4c55-8f1a-7d9c73e46994-kube-api-access-gmcmk\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891344    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-k8s-cni-cncf-io\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-netns\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891397    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgffw\" (UniqueName: \"kubernetes.io/projected/07790bc9-5df8-4bb9-8739-afe6fe8b9473-kube-api-access-zgffw\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8xkg\" (UniqueName: \"kubernetes.io/projected/d9397095-0672-4769-8a7b-d21de088e931-kube-api-access-s8xkg\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891446    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-hostroot\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-kubelet\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891507    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-etc-kubernetes\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891519    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovnkube-script-lib\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.892010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.891583    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892070    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-systemd\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892197    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-system-cni-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892458    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-conf-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892504    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-cni-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892536    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-cnibin\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892592    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/07790bc9-5df8-4bb9-8739-afe6fe8b9473-host-slash\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.892648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892619    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-daemon-config\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892662    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-registration-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-systemd-units\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892731    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-slash\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892759    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-cni-netd\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892794    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysconfig\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892824    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9397095-0672-4769-8a7b-d21de088e931-etc-tuned\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892872    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lw4bw\" (UniqueName: \"kubernetes.io/projected/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-kube-api-access-lw4bw\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892900    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892957    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4f8x4\" (UniqueName: \"kubernetes.io/projected/b609cbed-1144-4c46-b9b8-aada77dd2cd9-kube-api-access-4f8x4\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892967    2577 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892985    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9397095-0672-4769-8a7b-d21de088e931-tmp\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893014    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-multus-certs\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893040    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-node-log\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893071    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893100    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.893138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893130    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d4644fb-55db-4320-9d71-ca7b885427f5-kubelet-config\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893159    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893189    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-cni-binary-copy\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893226    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-socket-dir-parent\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-daemon-config\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893253    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-lib-modules\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/04cdccb3-c41a-423d-8a73-ffd42a5ccaa9-konnectivity-ca\") pod \"konnectivity-agent-5d7px\" (UID: \"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9\") " pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893283    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-cni-bin\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.892925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysctl-conf\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893360    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-socket-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-kubelet-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893530    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-k8s-cni-cncf-io\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893552    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cni-binary-copy\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893572    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-host\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893608    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-cnibin\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893664    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-modprobe-d\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.893695    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:38.893982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893706    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-run\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.893756    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs podName:08fb1852-cdb8-4abb-97f6-f85fdb8f4651 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:39.393735556 +0000 UTC m=+3.007862440 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs") pod "network-metrics-daemon-qpkv8" (UID: "08fb1852-cdb8-4abb-97f6-f85fdb8f4651") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-serviceca\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.893904    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-netns\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-kubelet\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894120    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysconfig\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-etc-selinux\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894255    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-cni-dir\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894287    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-system-cni-dir\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-sysctl-d\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894350    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/c0726f3e-2027-4c55-8f1a-7d9c73e46994-hosts-file\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894378    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-log-socket\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894388    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-host\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-run-multus-certs\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894433    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-kubernetes\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894459    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0726f3e-2027-4c55-8f1a-7d9c73e46994-tmp-dir\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894472    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-cni-bin\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.895582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffxrv\" (UniqueName: \"kubernetes.io/projected/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-kube-api-access-ffxrv\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894512    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-var-lib-kubelet\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-cni-multus\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894544    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cnibin\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rggcm\" (UniqueName: \"kubernetes.io/projected/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-kube-api-access-rggcm\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894626    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-device-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894631    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-registration-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-ovn\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894670    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894686    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-cni-bin\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894744    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07790bc9-5df8-4bb9-8739-afe6fe8b9473-iptables-alerter-script\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894748    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-multus-socket-dir-parent\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894775    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-sys\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894912    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-device-dir\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894952    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-host-var-lib-cni-multus\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.894982    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-lib-modules\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.895136    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.896550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.895173    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-cni-binary-copy\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.897377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.895296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c0726f3e-2027-4c55-8f1a-7d9c73e46994-tmp-dir\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.897377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.895318    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-sys\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.897377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.895351    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/577e86c4-8d3c-448c-bc58-089749f835c5-sys-fs\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.897377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.895465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d9397095-0672-4769-8a7b-d21de088e931-etc-kubernetes\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.897377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.895481    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/07790bc9-5df8-4bb9-8739-afe6fe8b9473-iptables-alerter-script\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.897667 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.897383    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d9397095-0672-4769-8a7b-d21de088e931-etc-tuned\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.898501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.898111    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:32:38.898501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.898139    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:32:38.898501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.898150    2577 projected.go:196] Error preparing data for projected volume kube-api-access-m7fkm for pod openshift-network-diagnostics/network-check-target-dx6np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:38.898501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.898198    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm podName:6ef725d3-0848-48b2-ae9a-85a288104caa nodeName:}" failed. No retries permitted until 2026-09-10 15:32:39.398185278 +0000 UTC m=+3.012312175 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fkm" (UniqueName: "kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm") pod "network-check-target-dx6np" (UID: "6ef725d3-0848-48b2-ae9a-85a288104caa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:38.898501 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.898307    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/04cdccb3-c41a-423d-8a73-ffd42a5ccaa9-agent-certs\") pod \"konnectivity-agent-5d7px\" (UID: \"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9\") " pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:38.898501 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.898491    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d9397095-0672-4769-8a7b-d21de088e931-tmp\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.901476 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.901434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-thmzq\" (UniqueName: \"kubernetes.io/projected/e1ac5f46-ec24-4285-8753-95bbb3bc7d17-kube-api-access-thmzq\") pod \"multus-n52xc\" (UID: \"e1ac5f46-ec24-4285-8753-95bbb3bc7d17\") " pod="openshift-multus/multus-n52xc"
Sep 10 15:32:38.905075 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.904973    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgffw\" (UniqueName: \"kubernetes.io/projected/07790bc9-5df8-4bb9-8739-afe6fe8b9473-kube-api-access-zgffw\") pod \"iptables-alerter-8sc54\" (UID: \"07790bc9-5df8-4bb9-8739-afe6fe8b9473\") " pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:38.907977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.907190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lw4bw\" (UniqueName: \"kubernetes.io/projected/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-kube-api-access-lw4bw\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:38.907977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.907252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal" event={"ID":"c65e95aed64261e20e4361dcc2a37963","Type":"ContainerStarted","Data":"71b5c7ec3bd07022fd96997dba584285dda09619eacee9172a5b02022d6ee39f"}
Sep 10 15:32:38.907977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.907442    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffxrv\" (UniqueName: \"kubernetes.io/projected/d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac-kube-api-access-ffxrv\") pod \"multus-additional-cni-plugins-vzwtb\" (UID: \"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac\") " pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:38.907977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.907535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rggcm\" (UniqueName: \"kubernetes.io/projected/9bf31f82-2e50-4346-ba50-f4b6edcd9a2b-kube-api-access-rggcm\") pod \"node-ca-jbtrv\" (UID: \"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b\") " pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:38.907977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.907925    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8xkg\" (UniqueName: \"kubernetes.io/projected/d9397095-0672-4769-8a7b-d21de088e931-kube-api-access-s8xkg\") pod \"tuned-fb8nl\" (UID: \"d9397095-0672-4769-8a7b-d21de088e931\") " pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:38.908308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.908075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmcmk\" (UniqueName: \"kubernetes.io/projected/c0726f3e-2027-4c55-8f1a-7d9c73e46994-kube-api-access-gmcmk\") pod \"node-resolver-hm9kh\" (UID: \"c0726f3e-2027-4c55-8f1a-7d9c73e46994\") " pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:38.909368 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.909260    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal" event={"ID":"b183149671d7e63b663b38bcacd05dd5","Type":"ContainerStarted","Data":"d5d68373bb285fa09b07e467fe2e47719840110434df4713a4ebe32012bb0acc"}
Sep 10 15:32:38.909463 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.909452    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft9nf\" (UniqueName: \"kubernetes.io/projected/577e86c4-8d3c-448c-bc58-089749f835c5-kube-api-access-ft9nf\") pod \"aws-ebs-csi-driver-node-f9jdp\" (UID: \"577e86c4-8d3c-448c-bc58-089749f835c5\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995719    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovn-node-metrics-cert\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995766    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-run-netns\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995810    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-etc-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995836    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovnkube-config\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995903    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-etc-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995938    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovnkube-script-lib\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995940    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-run-netns\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.995979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-systemd-units\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-slash\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996031    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-cni-netd\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996050    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4f8x4\" (UniqueName: \"kubernetes.io/projected/b609cbed-1144-4c46-b9b8-aada77dd2cd9-kube-api-access-4f8x4\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996074    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-node-log\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996090    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996123    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d4644fb-55db-4320-9d71-ca7b885427f5-kubelet-config\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996141    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-log-socket\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.996541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996160    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-ovn\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996174    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-cni-bin\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-kubelet\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996210    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-systemd\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-var-lib-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996266    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-run-ovn-kubernetes\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996281    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-env-overrides\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996309    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d4644fb-55db-4320-9d71-ca7b885427f5-dbus\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.996335    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.996457    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:38.996522    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret podName:5d4644fb-55db-4320-9d71-ca7b885427f5 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:39.496502995 +0000 UTC m=+3.110629879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret") pod "global-pull-secret-syncer-j5nr8" (UID: "5d4644fb-55db-4320-9d71-ca7b885427f5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997053    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovnkube-config\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovnkube-script-lib\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997341    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-node-log\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997447    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-systemd-units\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.997527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997500    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/5d4644fb-55db-4320-9d71-ca7b885427f5-dbus\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997498    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-slash\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-cni-netd\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997623    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-cni-bin\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997632    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/5d4644fb-55db-4320-9d71-ca7b885427f5-kubelet-config\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997679    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997695    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-log-socket\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997732    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-systemd\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997767    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-ovn\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-host-kubelet\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-var-lib-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.997820    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/b609cbed-1144-4c46-b9b8-aada77dd2cd9-run-openvswitch\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.998090    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/b609cbed-1144-4c46-b9b8-aada77dd2cd9-env-overrides\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:38.998979 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:38.998961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/b609cbed-1144-4c46-b9b8-aada77dd2cd9-ovn-node-metrics-cert\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:39.006707 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.006638    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4f8x4\" (UniqueName: \"kubernetes.io/projected/b609cbed-1144-4c46-b9b8-aada77dd2cd9-kube-api-access-4f8x4\") pod \"ovnkube-node-dk4jq\" (UID: \"b609cbed-1144-4c46-b9b8-aada77dd2cd9\") " pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:39.089798 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.089748    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fb8nl"
Sep 10 15:32:39.096933 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.096907    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-8sc54"
Sep 10 15:32:39.107824 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.107802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hm9kh"
Sep 10 15:32:39.118806 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.118775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vzwtb"
Sep 10 15:32:39.126720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.126694    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-n52xc"
Sep 10 15:32:39.135411 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.135389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jbtrv"
Sep 10 15:32:39.144087 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.144063    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:39.152966 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.152942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp"
Sep 10 15:32:39.160972 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.160950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:32:39.184410 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.184387    2577 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:32:39.399161 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.399088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:39.399161 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.399138    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:39.399400 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.399264    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:32:39.399400 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.399284    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:32:39.399400 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.399283    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:39.399400 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.399293    2577 projected.go:196] Error preparing data for projected volume kube-api-access-m7fkm for pod openshift-network-diagnostics/network-check-target-dx6np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:39.399400 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.399351    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs podName:08fb1852-cdb8-4abb-97f6-f85fdb8f4651 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:40.399333785 +0000 UTC m=+4.013460673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs") pod "network-metrics-daemon-qpkv8" (UID: "08fb1852-cdb8-4abb-97f6-f85fdb8f4651") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:39.399400 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.399371    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm podName:6ef725d3-0848-48b2-ae9a-85a288104caa nodeName:}" failed. No retries permitted until 2026-09-10 15:32:40.399361091 +0000 UTC m=+4.013487978 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fkm" (UniqueName: "kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm") pod "network-check-target-dx6np" (UID: "6ef725d3-0848-48b2-ae9a-85a288104caa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:39.500455 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.500423    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:39.500641 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.500576    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:39.500705 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:39.500645    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret podName:5d4644fb-55db-4320-9d71-ca7b885427f5 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:40.500628368 +0000 UTC m=+4.114755267 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret") pod "global-pull-secret-syncer-j5nr8" (UID: "5d4644fb-55db-4320-9d71-ca7b885427f5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:39.826527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.826422    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:27:39 +0000 UTC" deadline="2028-02-26 19:25:41.23984312 +0000 UTC"
Sep 10 15:32:39.826527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:39.826473    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12819h53m1.413375375s"
Sep 10 15:32:40.164809 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.164773    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0726f3e_2027_4c55_8f1a_7d9c73e46994.slice/crio-875b437a4ba3b1549ca6d7ce1313c5c4d7b40ff40593e5d7f8025849f09cda67 WatchSource:0}: Error finding container 875b437a4ba3b1549ca6d7ce1313c5c4d7b40ff40593e5d7f8025849f09cda67: Status 404 returned error can't find the container with id 875b437a4ba3b1549ca6d7ce1313c5c4d7b40ff40593e5d7f8025849f09cda67
Sep 10 15:32:40.167457 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.167358    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1ac5f46_ec24_4285_8753_95bbb3bc7d17.slice/crio-cde84f45c9575522c4648d9695fc7d563661ee126b168d12ee4e9fb5dadc20f7 WatchSource:0}: Error finding container cde84f45c9575522c4648d9695fc7d563661ee126b168d12ee4e9fb5dadc20f7: Status 404 returned error can't find the container with id cde84f45c9575522c4648d9695fc7d563661ee126b168d12ee4e9fb5dadc20f7
Sep 10 15:32:40.171464 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.171430    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577e86c4_8d3c_448c_bc58_089749f835c5.slice/crio-6c9b82f085926e900e67e31c0bdccf6e1ed06bf9a812181ed74e12be46cb161b WatchSource:0}: Error finding container 6c9b82f085926e900e67e31c0bdccf6e1ed06bf9a812181ed74e12be46cb161b: Status 404 returned error can't find the container with id 6c9b82f085926e900e67e31c0bdccf6e1ed06bf9a812181ed74e12be46cb161b
Sep 10 15:32:40.172482 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.172458    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07790bc9_5df8_4bb9_8739_afe6fe8b9473.slice/crio-4803350dc035090faf78aeef802d5cbbfe5b0461c5f3f402522aef99097ec7cd WatchSource:0}: Error finding container 4803350dc035090faf78aeef802d5cbbfe5b0461c5f3f402522aef99097ec7cd: Status 404 returned error can't find the container with id 4803350dc035090faf78aeef802d5cbbfe5b0461c5f3f402522aef99097ec7cd
Sep 10 15:32:40.174400 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.174361    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bf31f82_2e50_4346_ba50_f4b6edcd9a2b.slice/crio-5d12772e25f8aaa36f03ef55ae8034e9259376b69f1b450d3f1122d013839c47 WatchSource:0}: Error finding container 5d12772e25f8aaa36f03ef55ae8034e9259376b69f1b450d3f1122d013839c47: Status 404 returned error can't find the container with id 5d12772e25f8aaa36f03ef55ae8034e9259376b69f1b450d3f1122d013839c47
Sep 10 15:32:40.175348 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.175323    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb609cbed_1144_4c46_b9b8_aada77dd2cd9.slice/crio-b885772351b8b1f1e03750cc6a5fd53828504f0f2cb4c57d31f99018d5462fb8 WatchSource:0}: Error finding container b885772351b8b1f1e03750cc6a5fd53828504f0f2cb4c57d31f99018d5462fb8: Status 404 returned error can't find the container with id b885772351b8b1f1e03750cc6a5fd53828504f0f2cb4c57d31f99018d5462fb8
Sep 10 15:32:40.178022 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.177997    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e67c7f_d563_43d1_96b7_1cdeb9f7c1ac.slice/crio-f924f807d4d57ae7e97a910aa3a826d12526dc7d6991a239ca7b4e720af11d5b WatchSource:0}: Error finding container f924f807d4d57ae7e97a910aa3a826d12526dc7d6991a239ca7b4e720af11d5b: Status 404 returned error can't find the container with id f924f807d4d57ae7e97a910aa3a826d12526dc7d6991a239ca7b4e720af11d5b
Sep 10 15:32:40.179038 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.179011    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04cdccb3_c41a_423d_8a73_ffd42a5ccaa9.slice/crio-4ac34a8e01c00518b1b1d3d4bc75d9b7a983e5e2cd0c5d57e809539d9522ad86 WatchSource:0}: Error finding container 4ac34a8e01c00518b1b1d3d4bc75d9b7a983e5e2cd0c5d57e809539d9522ad86: Status 404 returned error can't find the container with id 4ac34a8e01c00518b1b1d3d4bc75d9b7a983e5e2cd0c5d57e809539d9522ad86
Sep 10 15:32:40.180748 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:32:40.180628    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9397095_0672_4769_8a7b_d21de088e931.slice/crio-c40eb4ec38fbc1859d1d185090fdd25403d393513829c9d0aa6486707f8dcc1b WatchSource:0}: Error finding container c40eb4ec38fbc1859d1d185090fdd25403d393513829c9d0aa6486707f8dcc1b: Status 404 returned error can't find the container with id c40eb4ec38fbc1859d1d185090fdd25403d393513829c9d0aa6486707f8dcc1b
Sep 10 15:32:40.406473 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.406437    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:40.406623 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.406484    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:40.406623 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.406604    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:40.406749 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.406637    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:32:40.406749 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.406664    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:32:40.406749 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.406669    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs podName:08fb1852-cdb8-4abb-97f6-f85fdb8f4651 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:42.406655499 +0000 UTC m=+6.020782382 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs") pod "network-metrics-daemon-qpkv8" (UID: "08fb1852-cdb8-4abb-97f6-f85fdb8f4651") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:40.406749 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.406678    2577 projected.go:196] Error preparing data for projected volume kube-api-access-m7fkm for pod openshift-network-diagnostics/network-check-target-dx6np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:40.406749 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.406734    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm podName:6ef725d3-0848-48b2-ae9a-85a288104caa nodeName:}" failed. No retries permitted until 2026-09-10 15:32:42.406716062 +0000 UTC m=+6.020842948 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fkm" (UniqueName: "kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm") pod "network-check-target-dx6np" (UID: "6ef725d3-0848-48b2-ae9a-85a288104caa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:40.507139 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.507107    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:40.507269 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.507253    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:40.507327 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.507319    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret podName:5d4644fb-55db-4320-9d71-ca7b885427f5 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:42.507303357 +0000 UTC m=+6.121430256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret") pod "global-pull-secret-syncer-j5nr8" (UID: "5d4644fb-55db-4320-9d71-ca7b885427f5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:40.827055 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.826972    2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:27:39 +0000 UTC" deadline="2028-05-01 03:07:43.206978823 +0000 UTC"
Sep 10 15:32:40.827055 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.827012    2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14363h35m2.379971875s"
Sep 10 15:32:40.904668 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.904634    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:40.904859 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.904788    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:40.905257 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.905234    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:40.905360 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.905336    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:40.905428 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.905409    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:40.905504 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:40.905485    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:40.921471 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.921431    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sc54" event={"ID":"07790bc9-5df8-4bb9-8739-afe6fe8b9473","Type":"ContainerStarted","Data":"4803350dc035090faf78aeef802d5cbbfe5b0461c5f3f402522aef99097ec7cd"}
Sep 10 15:32:40.926978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.926949    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n52xc" event={"ID":"e1ac5f46-ec24-4285-8753-95bbb3bc7d17","Type":"ContainerStarted","Data":"cde84f45c9575522c4648d9695fc7d563661ee126b168d12ee4e9fb5dadc20f7"}
Sep 10 15:32:40.935095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.935010    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hm9kh" event={"ID":"c0726f3e-2027-4c55-8f1a-7d9c73e46994","Type":"ContainerStarted","Data":"875b437a4ba3b1549ca6d7ce1313c5c4d7b40ff40593e5d7f8025849f09cda67"}
Sep 10 15:32:40.938853 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.938795    2577 generic.go:356] "Generic (PLEG): container finished" podID="c65e95aed64261e20e4361dcc2a37963" containerID="3ad0e273130e5498965c7c25f2e5a0c3c9dd4706cc59076c7d74787b38b2af9a" exitCode=0
Sep 10 15:32:40.938984 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.938898    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal" event={"ID":"c65e95aed64261e20e4361dcc2a37963","Type":"ContainerDied","Data":"3ad0e273130e5498965c7c25f2e5a0c3c9dd4706cc59076c7d74787b38b2af9a"}
Sep 10 15:32:40.947195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.947154    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fb8nl" event={"ID":"d9397095-0672-4769-8a7b-d21de088e931","Type":"ContainerStarted","Data":"c40eb4ec38fbc1859d1d185090fdd25403d393513829c9d0aa6486707f8dcc1b"}
Sep 10 15:32:40.956508 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.956454    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5d7px" event={"ID":"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9","Type":"ContainerStarted","Data":"4ac34a8e01c00518b1b1d3d4bc75d9b7a983e5e2cd0c5d57e809539d9522ad86"}
Sep 10 15:32:40.958924 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.958881    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerStarted","Data":"f924f807d4d57ae7e97a910aa3a826d12526dc7d6991a239ca7b4e720af11d5b"}
Sep 10 15:32:40.963751 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.963643    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp" event={"ID":"577e86c4-8d3c-448c-bc58-089749f835c5","Type":"ContainerStarted","Data":"6c9b82f085926e900e67e31c0bdccf6e1ed06bf9a812181ed74e12be46cb161b"}
Sep 10 15:32:40.976028 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.975996    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal" event={"ID":"b183149671d7e63b663b38bcacd05dd5","Type":"ContainerStarted","Data":"a86ec3bec217fc406d4fd3ce950367d16a8e9fbf6f7bc1048826ad41e6904f82"}
Sep 10 15:32:40.978987 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.978962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"b885772351b8b1f1e03750cc6a5fd53828504f0f2cb4c57d31f99018d5462fb8"}
Sep 10 15:32:40.987468 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.987435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbtrv" event={"ID":"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b","Type":"ContainerStarted","Data":"5d12772e25f8aaa36f03ef55ae8034e9259376b69f1b450d3f1122d013839c47"}
Sep 10 15:32:40.992603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:40.992552    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-11.ec2.internal" podStartSLOduration=2.992530528 podStartE2EDuration="2.992530528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:40.992237986 +0000 UTC m=+4.606364894" watchObservedRunningTime="2026-09-10 15:32:40.992530528 +0000 UTC m=+4.606657436"
Sep 10 15:32:42.019985 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:42.019945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal" event={"ID":"c65e95aed64261e20e4361dcc2a37963","Type":"ContainerStarted","Data":"a69892a921e716da8bff8edc6f780f707d7d50d6f84356a00acd5a9b6c9238d3"}
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:42.425114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:42.425203    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.425362    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.425380    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.425393    2577 projected.go:196] Error preparing data for projected volume kube-api-access-m7fkm for pod openshift-network-diagnostics/network-check-target-dx6np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.425450    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm podName:6ef725d3-0848-48b2-ae9a-85a288104caa nodeName:}" failed. No retries permitted until 2026-09-10 15:32:46.425432035 +0000 UTC m=+10.039558923 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fkm" (UniqueName: "kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm") pod "network-check-target-dx6np" (UID: "6ef725d3-0848-48b2-ae9a-85a288104caa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.425943    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:42.427103 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.426012    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs podName:08fb1852-cdb8-4abb-97f6-f85fdb8f4651 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:46.425995693 +0000 UTC m=+10.040122578 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs") pod "network-metrics-daemon-qpkv8" (UID: "08fb1852-cdb8-4abb-97f6-f85fdb8f4651") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:42.526486 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:42.526444    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:42.526748 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.526653    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:42.526748 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.526722    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret podName:5d4644fb-55db-4320-9d71-ca7b885427f5 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:46.526701947 +0000 UTC m=+10.140828863 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret") pod "global-pull-secret-syncer-j5nr8" (UID: "5d4644fb-55db-4320-9d71-ca7b885427f5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:42.904896 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:42.904797    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:42.905063 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.904971    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:42.905395 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:42.905374    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:42.905494 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.905475    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:42.905552 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:42.905538    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:42.905654 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:42.905615    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:44.904560 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:44.903939    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:44.904560 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:44.904070    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:44.904560 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:44.904127    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:44.904560 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:44.904210    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:44.904560 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:44.904261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:44.904560 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:44.904348    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:46.461512 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:46.461262    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:46.461991 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.461548    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:32:46.461991 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:46.461551    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:46.461991 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.461574    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:32:46.461991 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.461589    2577 projected.go:196] Error preparing data for projected volume kube-api-access-m7fkm for pod openshift-network-diagnostics/network-check-target-dx6np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:46.461991 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.461651    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm podName:6ef725d3-0848-48b2-ae9a-85a288104caa nodeName:}" failed. No retries permitted until 2026-09-10 15:32:54.461630116 +0000 UTC m=+18.075757014 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fkm" (UniqueName: "kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm") pod "network-check-target-dx6np" (UID: "6ef725d3-0848-48b2-ae9a-85a288104caa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:46.461991 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.461676    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:46.461991 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.461729    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs podName:08fb1852-cdb8-4abb-97f6-f85fdb8f4651 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:54.461712919 +0000 UTC m=+18.075839806 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs") pod "network-metrics-daemon-qpkv8" (UID: "08fb1852-cdb8-4abb-97f6-f85fdb8f4651") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:46.561883 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:46.561834    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:46.562044 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.561981    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:46.562044 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.562031    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret podName:5d4644fb-55db-4320-9d71-ca7b885427f5 nodeName:}" failed. No retries permitted until 2026-09-10 15:32:54.562015558 +0000 UTC m=+18.176142446 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret") pod "global-pull-secret-syncer-j5nr8" (UID: "5d4644fb-55db-4320-9d71-ca7b885427f5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:46.903036 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:46.902998    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:46.903259 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.903120    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:46.903699 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:46.903543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:46.903699 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.903654    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:46.903908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:46.903824    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:46.903969 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:46.903951    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:48.903425 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:48.903390    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:48.903905 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:48.903417    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:48.903905 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:48.903543    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:48.903905 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:48.903676    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:48.903905 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:48.903801    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:48.903905 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:48.903529    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:50.902386 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:50.902347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:50.902845 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:50.902354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:50.902845 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:50.902482    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:50.902845 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:50.902478    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:50.902845 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:50.902564    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:50.902845 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:50.902667    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:52.902975 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:52.902494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:52.902975 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:52.902631    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:52.902975 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:52.902640    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:52.902975 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:52.902660    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:52.902975 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:52.902737    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:52.902975 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:52.902836    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:54.527977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:54.527942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:54.528470 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:54.527996    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:54.528470 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.528112    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:54.528470 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.528114    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:32:54.528470 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.528142    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:32:54.528470 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.528152    2577 projected.go:196] Error preparing data for projected volume kube-api-access-m7fkm for pod openshift-network-diagnostics/network-check-target-dx6np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:54.528470 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.528183    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs podName:08fb1852-cdb8-4abb-97f6-f85fdb8f4651 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:10.528163223 +0000 UTC m=+34.142290118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs") pod "network-metrics-daemon-qpkv8" (UID: "08fb1852-cdb8-4abb-97f6-f85fdb8f4651") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:32:54.528470 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.528206    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm podName:6ef725d3-0848-48b2-ae9a-85a288104caa nodeName:}" failed. No retries permitted until 2026-09-10 15:33:10.528190031 +0000 UTC m=+34.142316937 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fkm" (UniqueName: "kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm") pod "network-check-target-dx6np" (UID: "6ef725d3-0848-48b2-ae9a-85a288104caa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:32:54.629143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:54.629096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:54.629331 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.629240    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:54.629331 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.629320    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret podName:5d4644fb-55db-4320-9d71-ca7b885427f5 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:10.629298786 +0000 UTC m=+34.243425672 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret") pod "global-pull-secret-syncer-j5nr8" (UID: "5d4644fb-55db-4320-9d71-ca7b885427f5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:32:54.902362 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:54.902331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:54.902362 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:54.902354    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:54.902579 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:54.902353    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:54.902579 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.902459    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:54.902579 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.902516    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:54.902672 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:54.902595    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:56.902812 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:56.902775    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:56.903289 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:56.902869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:56.903289 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:56.902880    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:56.903289 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:56.902969    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:56.903289 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:56.902966    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:56.903289 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:56.903049    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:58.902342 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:58.902267    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:32:58.902777 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:58.902389    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:32:58.902777 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:58.902736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:32:58.902894 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:58.902860    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:32:58.902954 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:58.902921    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:32:58.903006 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:32:58.902993    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:32:59.063486 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.063451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hm9kh" event={"ID":"c0726f3e-2027-4c55-8f1a-7d9c73e46994","Type":"ContainerStarted","Data":"1d875e88648386112470bbc5dc4149f950242b5b384e322ae186969c482fbce8"}
Sep 10 15:32:59.067476 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.067445    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fb8nl" event={"ID":"d9397095-0672-4769-8a7b-d21de088e931","Type":"ContainerStarted","Data":"3ab95600eedbd827caa34b98ab3c6f96ef9ceb0f8ee3842bfc95ad890709881c"}
Sep 10 15:32:59.071367 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.071338    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-5d7px" event={"ID":"04cdccb3-c41a-423d-8a73-ffd42a5ccaa9","Type":"ContainerStarted","Data":"5aec317a8b7b5cdacbfc01ea9d1218466f9cf2d4f788c42e3cc648d6793ecf3c"}
Sep 10 15:32:59.079264 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.075868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-n52xc" event={"ID":"e1ac5f46-ec24-4285-8753-95bbb3bc7d17","Type":"ContainerStarted","Data":"fe12c42849da6a051107ffae5c7176e6dc3a55c0d4e435bfcc00bf89fdb7697f"}
Sep 10 15:32:59.079264 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.077377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-11.ec2.internal" podStartSLOduration=21.07736529 podStartE2EDuration="21.07736529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:42.037589941 +0000 UTC m=+5.651716851" watchObservedRunningTime="2026-09-10 15:32:59.07736529 +0000 UTC m=+22.691492195"
Sep 10 15:32:59.098373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.098040    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-hm9kh" podStartSLOduration=3.393047055 podStartE2EDuration="22.098018329s" totalImagesPullingTime="18.704971274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:59.077903695 +0000 UTC m=+22.692030598" watchObservedRunningTime="2026-09-10 15:32:59.098018329 +0000 UTC m=+22.712145237"
Sep 10 15:32:59.098373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.098323    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-n52xc" podStartSLOduration=3.327374834 podStartE2EDuration="22.098313164s" totalImagesPullingTime="18.77093833s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:59.097801814 +0000 UTC m=+22.711928739" watchObservedRunningTime="2026-09-10 15:32:59.098313164 +0000 UTC m=+22.712440069"
Sep 10 15:32:59.114827 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.114769    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-5d7px" podStartSLOduration=2.426796983 podStartE2EDuration="21.114731333s" totalImagesPullingTime="18.68793435s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:59.114297705 +0000 UTC m=+22.728424610" watchObservedRunningTime="2026-09-10 15:32:59.114731333 +0000 UTC m=+22.728858238"
Sep 10 15:32:59.130669 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.130516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fb8nl" podStartSLOduration=3.396609771 podStartE2EDuration="22.130493527s" totalImagesPullingTime="18.733883756s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:32:59.129957104 +0000 UTC m=+22.744084010" watchObservedRunningTime="2026-09-10 15:32:59.130493527 +0000 UTC m=+22.744620509"
Sep 10 15:32:59.910566 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.910380    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:32:59.911354 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:32:59.911186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:33:00.080025 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.079996    2577 generic.go:356] "Generic (PLEG): container finished" podID="d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac" containerID="a0ac16694e9b93a00fecc64796946733f4c6ef63861e15b2df733bf1757ebc50" exitCode=0
Sep 10 15:33:00.080194 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.080072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerDied","Data":"a0ac16694e9b93a00fecc64796946733f4c6ef63861e15b2df733bf1757ebc50"}
Sep 10 15:33:00.081737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.081528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp" event={"ID":"577e86c4-8d3c-448c-bc58-089749f835c5","Type":"ContainerStarted","Data":"501d1cf07ae577604c030b8309f15e13cc56c81b84d1c8fc3487fae7d156a195"}
Sep 10 15:33:00.084666 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.084646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"f31aaf79ee9a9634abc20cfd1e5f750dac3041cd154b5094a5cc3ecee0b19acb"}
Sep 10 15:33:00.084791 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.084700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"aa7a60daf70f77573f3e1288355180904856ecf2702304bf4be7cb5acc3916cc"}
Sep 10 15:33:00.084791 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.084739    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"0258fbf5b807abddf2d1dcbf1ec44dcad00408829f37cdbb203e578f42648739"}
Sep 10 15:33:00.084962 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.084799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"ef5dd441153d52f81fd3694bd2eaa756f231d6b49f8caa78dced7c6d0fc7dcd7"}
Sep 10 15:33:00.084962 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.084846    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"6e9aa7c1ac037646f4d8b247237602c0374fc9d41fe34b2752072b9ff83c92c6"}
Sep 10 15:33:00.084962 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.084945    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"048f238f850e7c96f0208766f1948ddecde22990787a53b131d12034580c2ef6"}
Sep 10 15:33:00.086091 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.086058    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jbtrv" event={"ID":"9bf31f82-2e50-4346-ba50-f4b6edcd9a2b","Type":"ContainerStarted","Data":"8b7827f813615411037e87520df6ed84ab95aa1b394b527e4f8804a16703a38d"}
Sep 10 15:33:00.114611 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.114570    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jbtrv" podStartSLOduration=4.420559734 podStartE2EDuration="23.114558258s" totalImagesPullingTime="18.693998524s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:00.114484591 +0000 UTC m=+23.728611495" watchObservedRunningTime="2026-09-10 15:33:00.114558258 +0000 UTC m=+23.728685162"
Sep 10 15:33:00.428723 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.428556    2577 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 15:33:00.804919 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.804888    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:33:00.805515 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.805430    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-5d7px"
Sep 10 15:33:00.852837 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.852725    2577 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:33:00.428710458Z","UUID":"2df41ec7-9f22-462d-99ce-30f3ed4ac822","Handler":null,"Name":"","Endpoint":""}
Sep 10 15:33:00.854946 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.854920    2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 15:33:00.855077 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.854958    2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 15:33:00.901826 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.901764    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:00.901826 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.901802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:00.902050 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:00.901767    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:00.902050 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:00.901889    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:33:00.902050 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:00.902006    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:33:00.902181 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:00.902101    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:33:01.091304 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:01.091263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp" event={"ID":"577e86c4-8d3c-448c-bc58-089749f835c5","Type":"ContainerStarted","Data":"99a2f5ca1b42fe505c41109e9e0f764b26cf7a2c2111c8231c973d904ab7c883"}
Sep 10 15:33:01.092858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:01.092828    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-8sc54" event={"ID":"07790bc9-5df8-4bb9-8739-afe6fe8b9473","Type":"ContainerStarted","Data":"4bf2b8ebdedc28a8f1eee1d04246088716915e3914726988c7071854b5fde18e"}
Sep 10 15:33:01.108021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:01.107969    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-8sc54" podStartSLOduration=5.412937182 podStartE2EDuration="24.107951637s" totalImagesPullingTime="18.695014455s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:01.107565383 +0000 UTC m=+24.721692308" watchObservedRunningTime="2026-09-10 15:33:01.107951637 +0000 UTC m=+24.722078545"
Sep 10 15:33:02.097720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:02.097680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp" event={"ID":"577e86c4-8d3c-448c-bc58-089749f835c5","Type":"ContainerStarted","Data":"21908d5bb1c2abf6a34f7a8ed264c65b0022ba49b2f2097d938f5205845fed8c"}
Sep 10 15:33:02.101895 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:02.101868    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"ab78ab6bb9f494f503d42ab8d9c89947e46aaec0f3821ab70ff2445868555055"}
Sep 10 15:33:02.119144 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:02.119101    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-f9jdp" podStartSLOduration=3.316202043 podStartE2EDuration="24.119085313s" totalImagesPullingTime="20.80288327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:02.118971128 +0000 UTC m=+25.733098034" watchObservedRunningTime="2026-09-10 15:33:02.119085313 +0000 UTC m=+25.733212217"
Sep 10 15:33:02.902545 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:02.902508    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:02.902722 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:02.902556    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:02.902722 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:02.902511    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:02.902722 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:02.902645    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:33:02.902919 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:02.902736    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:33:02.902919 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:02.902821    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:33:04.902777 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:04.902527    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:04.903204 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:04.902529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:04.903204 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:04.902835    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:33:04.903204 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:04.902529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:04.903204 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:04.902892    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:33:04.903204 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:04.902954    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:33:05.112814 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:05.112775    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" event={"ID":"b609cbed-1144-4c46-b9b8-aada77dd2cd9","Type":"ContainerStarted","Data":"112bef7ad6921dcd9450d964c18f7b24b897108dbb54137d8e2b01c567eb83b0"}
Sep 10 15:33:05.113216 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:05.113186    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:33:05.113330 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:05.113261    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:33:05.114864 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:05.114843    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerStarted","Data":"f232cdbe85fd3de6f844c1c51b6d57bb1afd3884d056ef46003eee31fade7648"}
Sep 10 15:33:05.128557 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:05.128538    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:33:05.148203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:05.148161    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq" podStartSLOduration=8.363659036 podStartE2EDuration="27.14814823s" totalImagesPullingTime="18.784489194s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:05.145187969 +0000 UTC m=+28.759314873" watchObservedRunningTime="2026-09-10 15:33:05.14814823 +0000 UTC m=+28.762275134"
Sep 10 15:33:06.119436 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.119397    2577 generic.go:356] "Generic (PLEG): container finished" podID="d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac" containerID="f232cdbe85fd3de6f844c1c51b6d57bb1afd3884d056ef46003eee31fade7648" exitCode=0
Sep 10 15:33:06.120926 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.120901    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerDied","Data":"f232cdbe85fd3de6f844c1c51b6d57bb1afd3884d056ef46003eee31fade7648"}
Sep 10 15:33:06.121061 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.121025    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:33:06.144228 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.144202    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:33:06.611232 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.611164    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dx6np"]
Sep 10 15:33:06.611421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.611402    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:06.611569 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:06.611523    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:33:06.611875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.611847    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j5nr8"]
Sep 10 15:33:06.611979 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.611958    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:06.612087 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:06.612065    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:33:06.613983 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.613952    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qpkv8"]
Sep 10 15:33:06.614178 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:06.614054    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:06.614295 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:06.614271    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:33:06.881111 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:06.880886    2577 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e67c7f_d563_43d1_96b7_1cdeb9f7c1ac.slice/crio-e383c92609d2d6c7f4dad3f5989b1b786bb3b535c425741f3d93aa5eec3325cf.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1e67c7f_d563_43d1_96b7_1cdeb9f7c1ac.slice/crio-conmon-e383c92609d2d6c7f4dad3f5989b1b786bb3b535c425741f3d93aa5eec3325cf.scope\": RecentStats: unable to find data in memory cache]"
Sep 10 15:33:07.124168 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:07.124133    2577 generic.go:356] "Generic (PLEG): container finished" podID="d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac" containerID="e383c92609d2d6c7f4dad3f5989b1b786bb3b535c425741f3d93aa5eec3325cf" exitCode=0
Sep 10 15:33:07.124622 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:07.124217    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerDied","Data":"e383c92609d2d6c7f4dad3f5989b1b786bb3b535c425741f3d93aa5eec3325cf"}
Sep 10 15:33:07.901646 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:07.901610    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:07.901829 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:07.901619    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:07.901829 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:07.901767    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:33:07.901946 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:07.901903    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:33:08.129343 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:08.129313    2577 generic.go:356] "Generic (PLEG): container finished" podID="d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac" containerID="c764d6ff641e9c99ace66cf696cfef8a8523007fd0bcf0d7934bcc70101c0379" exitCode=0
Sep 10 15:33:08.129739 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:08.129390    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerDied","Data":"c764d6ff641e9c99ace66cf696cfef8a8523007fd0bcf0d7934bcc70101c0379"}
Sep 10 15:33:08.901767 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:08.901721    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:08.901943 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:08.901866    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:33:09.902507 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:09.902471    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:09.902507 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:09.902494    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:09.902994 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:09.902608    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dx6np" podUID="6ef725d3-0848-48b2-ae9a-85a288104caa"
Sep 10 15:33:09.902994 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:09.902784    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpkv8" podUID="08fb1852-cdb8-4abb-97f6-f85fdb8f4651"
Sep 10 15:33:10.540201 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:10.540164    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:10.540464 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:10.540215    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:10.540464 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.540335    2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:33:10.540464 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.540348    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:33:10.540464 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.540364    2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:33:10.540464 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.540375    2577 projected.go:196] Error preparing data for projected volume kube-api-access-m7fkm for pod openshift-network-diagnostics/network-check-target-dx6np: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:33:10.540464 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.540403    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs podName:08fb1852-cdb8-4abb-97f6-f85fdb8f4651 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:42.540384053 +0000 UTC m=+66.154510940 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs") pod "network-metrics-daemon-qpkv8" (UID: "08fb1852-cdb8-4abb-97f6-f85fdb8f4651") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:33:10.540464 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.540420    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm podName:6ef725d3-0848-48b2-ae9a-85a288104caa nodeName:}" failed. No retries permitted until 2026-09-10 15:33:42.540411695 +0000 UTC m=+66.154538577 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7fkm" (UniqueName: "kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm") pod "network-check-target-dx6np" (UID: "6ef725d3-0848-48b2-ae9a-85a288104caa") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:33:10.641553 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:10.641511    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:10.641732 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.641642    2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:33:10.641732 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.641704    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret podName:5d4644fb-55db-4320-9d71-ca7b885427f5 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:42.641691268 +0000 UTC m=+66.255818151 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret") pod "global-pull-secret-syncer-j5nr8" (UID: "5d4644fb-55db-4320-9d71-ca7b885427f5") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:33:10.902404 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:10.902366    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:10.902598 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:10.902506    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-j5nr8" podUID="5d4644fb-55db-4320-9d71-ca7b885427f5"
Sep 10 15:33:11.335359 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.335332    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hm9kh_c0726f3e-2027-4c55-8f1a-7d9c73e46994/dns-node-resolver/0.log"
Sep 10 15:33:11.667528 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.667445    2577 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-11.ec2.internal" event="NodeReady"
Sep 10 15:33:11.667688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.667606    2577 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 15:33:11.714837 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.714798    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-h27xd"]
Sep 10 15:33:11.793211 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.793175    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dp7tl"]
Sep 10 15:33:11.793388 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.793307    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:11.795720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.795698    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 15:33:11.795869 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.795809    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 15:33:11.795869 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.795823    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 15:33:11.795869 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.795833    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 15:33:11.795869 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.795829    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cj5pp\""
Sep 10 15:33:11.815277 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.815254    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h27xd"]
Sep 10 15:33:11.815418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.815295    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dp7tl"]
Sep 10 15:33:11.815418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.815405    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:11.817847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.817808    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 15:33:11.817847 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.817816    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-bddbj\""
Sep 10 15:33:11.818018 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.817814    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 15:33:11.825653 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.825632    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-8zpnr"]
Sep 10 15:33:11.843060 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.843040    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zpnr"]
Sep 10 15:33:11.843196 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.843134    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:11.846092 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.845847    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 15:33:11.846092 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.845871    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-t4nnn\""
Sep 10 15:33:11.846092 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.845903    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 15:33:11.846092 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.845959    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 15:33:11.902015 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.901983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:11.902197 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.901983    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:11.904858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.904826    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:33:11.905235 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.905046    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:33:11.905235 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.905100    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gcptc\""
Sep 10 15:33:11.905235 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.905046    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:33:11.905235 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.905101    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skj7z\""
Sep 10 15:33:11.950676 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950599    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:11.950676 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950666    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ff4d54-bda4-480e-9f05-1b4b63e8fc08-cert\") pod \"ingress-canary-8zpnr\" (UID: \"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08\") " pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:11.950935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950727    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f195d304-9189-4b43-839c-25369d19155f-tmp-dir\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:11.950935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950791    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6lp7\" (UniqueName: \"kubernetes.io/projected/e5ff4d54-bda4-480e-9f05-1b4b63e8fc08-kube-api-access-s6lp7\") pod \"ingress-canary-8zpnr\" (UID: \"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08\") " pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:11.950935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950823    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:11.950935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vflfj\" (UniqueName: \"kubernetes.io/projected/38279413-0a92-4188-95cb-675c20eafd20-kube-api-access-vflfj\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:11.950935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950918    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38279413-0a92-4188-95cb-675c20eafd20-data-volume\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:11.951124 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f195d304-9189-4b43-839c-25369d19155f-config-volume\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:11.951124 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950971    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:11.951124 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.950992    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckp4c\" (UniqueName: \"kubernetes.io/projected/f195d304-9189-4b43-839c-25369d19155f-kube-api-access-ckp4c\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:11.951124 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:11.951024    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38279413-0a92-4188-95cb-675c20eafd20-crio-socket\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.051947 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.051908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f195d304-9189-4b43-839c-25369d19155f-tmp-dir\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.051947 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.051949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6lp7\" (UniqueName: \"kubernetes.io/projected/e5ff4d54-bda4-480e-9f05-1b4b63e8fc08-kube-api-access-s6lp7\") pod \"ingress-canary-8zpnr\" (UID: \"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08\") " pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:12.052199 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.051975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.052199 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052000    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vflfj\" (UniqueName: \"kubernetes.io/projected/38279413-0a92-4188-95cb-675c20eafd20-kube-api-access-vflfj\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.052199 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.052111    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:12.052199 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.052186    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:12.552163019 +0000 UTC m=+36.166289924 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:12.052444 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052208    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38279413-0a92-4188-95cb-675c20eafd20-data-volume\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.052444 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052240    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f195d304-9189-4b43-839c-25369d19155f-config-volume\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.052444 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052301    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f195d304-9189-4b43-839c-25369d19155f-tmp-dir\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.052444 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052342    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.052444 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052360    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckp4c\" (UniqueName: \"kubernetes.io/projected/f195d304-9189-4b43-839c-25369d19155f-kube-api-access-ckp4c\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.052444 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052378    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38279413-0a92-4188-95cb-675c20eafd20-crio-socket\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.052484    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.052535    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls podName:f195d304-9189-4b43-839c-25369d19155f nodeName:}" failed. No retries permitted until 2026-09-10 15:33:12.552520454 +0000 UTC m=+36.166647345 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls") pod "dns-default-dp7tl" (UID: "f195d304-9189-4b43-839c-25369d19155f") : secret "dns-default-metrics-tls" not found
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052538    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/38279413-0a92-4188-95cb-675c20eafd20-data-volume\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.052569    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052584    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ff4d54-bda4-480e-9f05-1b4b63e8fc08-cert\") pod \"ingress-canary-8zpnr\" (UID: \"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08\") " pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052587    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/38279413-0a92-4188-95cb-675c20eafd20-crio-socket\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.052775 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.052630    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:12.552612885 +0000 UTC m=+36.166739767 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : configmap "kube-rbac-proxy" not found
Sep 10 15:33:12.053094 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.052800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f195d304-9189-4b43-839c-25369d19155f-config-volume\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.057571 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.057411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e5ff4d54-bda4-480e-9f05-1b4b63e8fc08-cert\") pod \"ingress-canary-8zpnr\" (UID: \"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08\") " pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:12.062197 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.061883    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckp4c\" (UniqueName: \"kubernetes.io/projected/f195d304-9189-4b43-839c-25369d19155f-kube-api-access-ckp4c\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.062357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.062330    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vflfj\" (UniqueName: \"kubernetes.io/projected/38279413-0a92-4188-95cb-675c20eafd20-kube-api-access-vflfj\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.063202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.063161    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6lp7\" (UniqueName: \"kubernetes.io/projected/e5ff4d54-bda4-480e-9f05-1b4b63e8fc08-kube-api-access-s6lp7\") pod \"ingress-canary-8zpnr\" (UID: \"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08\") " pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:12.153159 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.153126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-8zpnr"
Sep 10 15:33:12.309611 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.309582    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbtrv_9bf31f82-2e50-4346-ba50-f4b6edcd9a2b/node-ca/0.log"
Sep 10 15:33:12.503977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.503945    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-8zpnr"]
Sep 10 15:33:12.506885 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:12.506863    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ff4d54_bda4_480e_9f05_1b4b63e8fc08.slice/crio-9d2a5f6949779bf4cc0df6bf6bcb722e6eaf39857eb3f6232ff3fce2d2c9d717 WatchSource:0}: Error finding container 9d2a5f6949779bf4cc0df6bf6bcb722e6eaf39857eb3f6232ff3fce2d2c9d717: Status 404 returned error can't find the container with id 9d2a5f6949779bf4cc0df6bf6bcb722e6eaf39857eb3f6232ff3fce2d2c9d717
Sep 10 15:33:12.557034 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.556999    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.557155 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.557072    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:12.557155 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.557124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:12.557315 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.557242    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:33:12.557364 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.557346    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls podName:f195d304-9189-4b43-839c-25369d19155f nodeName:}" failed. No retries permitted until 2026-09-10 15:33:13.557324753 +0000 UTC m=+37.171451635 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls") pod "dns-default-dp7tl" (UID: "f195d304-9189-4b43-839c-25369d19155f") : secret "dns-default-metrics-tls" not found
Sep 10 15:33:12.557597 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.557505    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:12.557597 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.557535    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:33:12.557597 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.557566    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:13.557551149 +0000 UTC m=+37.171678045 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:12.557830 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:12.557614    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:13.557595245 +0000 UTC m=+37.171722137 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : configmap "kube-rbac-proxy" not found
Sep 10 15:33:12.902237 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.902205    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:12.905153 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:12.905128    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:33:13.142108 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:13.142035    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zpnr" event={"ID":"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08","Type":"ContainerStarted","Data":"9d2a5f6949779bf4cc0df6bf6bcb722e6eaf39857eb3f6232ff3fce2d2c9d717"}
Sep 10 15:33:13.145593 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:13.145556    2577 generic.go:356] "Generic (PLEG): container finished" podID="d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac" containerID="625daabc50391cc57a1d1051e609276a57b7b22363db07a8f414633a02ff8894" exitCode=0
Sep 10 15:33:13.145720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:13.145591    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerDied","Data":"625daabc50391cc57a1d1051e609276a57b7b22363db07a8f414633a02ff8894"}
Sep 10 15:33:13.565298 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:13.565218    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:13.565298 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:13.565286    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:13.565501 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:13.565334    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:13.565501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:13.565369    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:33:13.565501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:13.565442    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:33:13.565501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:13.565449    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:13.565501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:13.565443    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:15.565421077 +0000 UTC m=+39.179547964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : configmap "kube-rbac-proxy" not found
Sep 10 15:33:13.565501 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:13.565503    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:15.565486582 +0000 UTC m=+39.179613469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:13.565719 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:13.565525    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls podName:f195d304-9189-4b43-839c-25369d19155f nodeName:}" failed. No retries permitted until 2026-09-10 15:33:15.565513715 +0000 UTC m=+39.179640602 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls") pod "dns-default-dp7tl" (UID: "f195d304-9189-4b43-839c-25369d19155f") : secret "dns-default-metrics-tls" not found
Sep 10 15:33:14.151215 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:14.151181    2577 generic.go:356] "Generic (PLEG): container finished" podID="d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac" containerID="34383008abddf1fd9a9ca422d2fdcbf01926aaaab93b2fe3491b19feaf9106b7" exitCode=0
Sep 10 15:33:14.151684 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:14.151241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerDied","Data":"34383008abddf1fd9a9ca422d2fdcbf01926aaaab93b2fe3491b19feaf9106b7"}
Sep 10 15:33:15.036005 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.035840    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vb9np"]
Sep 10 15:33:15.044202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.044174    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vb9np"]
Sep 10 15:33:15.044323 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.044282    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.047539 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.047501    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Sep 10 15:33:15.047539 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.047535    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Sep 10 15:33:15.047774 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.047583    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Sep 10 15:33:15.048466 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.048411    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Sep 10 15:33:15.048565 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.048548    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-pvz9l\""
Sep 10 15:33:15.156287 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.156200    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" event={"ID":"d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac","Type":"ContainerStarted","Data":"29d070ce9145409a46a75df2130e389057031243be866f58b7072623ea94d427"}
Sep 10 15:33:15.179097 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.179030    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d61e46a2-496f-4850-a406-bd254222a892-signing-key\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.179263 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.179173    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d61e46a2-496f-4850-a406-bd254222a892-signing-cabundle\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.179326 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.179281    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dncrv\" (UniqueName: \"kubernetes.io/projected/d61e46a2-496f-4850-a406-bd254222a892-kube-api-access-dncrv\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.180203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.180156    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vzwtb" podStartSLOduration=8.8857787 podStartE2EDuration="38.180142289s" totalImagesPullingTime="29.294363589s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:37 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:15.179256089 +0000 UTC m=+38.793382994" watchObservedRunningTime="2026-09-10 15:33:15.180142289 +0000 UTC m=+38.794269193"
Sep 10 15:33:15.280029 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.279989    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d61e46a2-496f-4850-a406-bd254222a892-signing-key\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.280225 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.280082    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d61e46a2-496f-4850-a406-bd254222a892-signing-cabundle\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.280225 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.280140    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dncrv\" (UniqueName: \"kubernetes.io/projected/d61e46a2-496f-4850-a406-bd254222a892-kube-api-access-dncrv\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.281012 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.280976    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/d61e46a2-496f-4850-a406-bd254222a892-signing-cabundle\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.284056 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.284027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/d61e46a2-496f-4850-a406-bd254222a892-signing-key\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.290120 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.290093    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dncrv\" (UniqueName: \"kubernetes.io/projected/d61e46a2-496f-4850-a406-bd254222a892-kube-api-access-dncrv\") pod \"service-ca-75d99c484-vb9np\" (UID: \"d61e46a2-496f-4850-a406-bd254222a892\") " pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.354095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.354050    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-75d99c484-vb9np"
Sep 10 15:33:15.582145 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.582106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:15.582282 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.582167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:15.582282 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:15.582226    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:33:15.582282 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:15.582273    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:15.582391 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:15.582293    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:33:15.582391 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:15.582314    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:19.582295701 +0000 UTC m=+43.196422583 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : configmap "kube-rbac-proxy" not found
Sep 10 15:33:15.582391 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:15.582331    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:19.582324765 +0000 UTC m=+43.196451647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:15.582391 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.582231    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:15.582391 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:15.582346    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls podName:f195d304-9189-4b43-839c-25369d19155f nodeName:}" failed. No retries permitted until 2026-09-10 15:33:19.582337523 +0000 UTC m=+43.196464411 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls") pod "dns-default-dp7tl" (UID: "f195d304-9189-4b43-839c-25369d19155f") : secret "dns-default-metrics-tls" not found
Sep 10 15:33:15.639927 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:15.639892    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-75d99c484-vb9np"]
Sep 10 15:33:15.643826 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:15.643796    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd61e46a2_496f_4850_a406_bd254222a892.slice/crio-fdac1c62e90483fa0ac6305d375cc1fdef0a3d4aebbd8c3af4f00ccdfc5d0c6b WatchSource:0}: Error finding container fdac1c62e90483fa0ac6305d375cc1fdef0a3d4aebbd8c3af4f00ccdfc5d0c6b: Status 404 returned error can't find the container with id fdac1c62e90483fa0ac6305d375cc1fdef0a3d4aebbd8c3af4f00ccdfc5d0c6b
Sep 10 15:33:16.159924 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:16.159892    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-8zpnr" event={"ID":"e5ff4d54-bda4-480e-9f05-1b4b63e8fc08","Type":"ContainerStarted","Data":"12e0427884095cedf71d124c32808d30f60aa355a94df0dd227d25f1815e41ff"}
Sep 10 15:33:16.161111 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:16.161090    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-vb9np" event={"ID":"d61e46a2-496f-4850-a406-bd254222a892","Type":"ContainerStarted","Data":"fdac1c62e90483fa0ac6305d375cc1fdef0a3d4aebbd8c3af4f00ccdfc5d0c6b"}
Sep 10 15:33:16.175978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:16.175936    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-8zpnr" podStartSLOduration=2.603908702 podStartE2EDuration="5.175917573s" totalImagesPullingTime="2.572008871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:16.175565528 +0000 UTC m=+39.789692432" watchObservedRunningTime="2026-09-10 15:33:16.175917573 +0000 UTC m=+39.790044469"
Sep 10 15:33:19.170147 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:19.169882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-75d99c484-vb9np" event={"ID":"d61e46a2-496f-4850-a406-bd254222a892","Type":"ContainerStarted","Data":"398ca87e9dd185adc2e153162b3619187f20f0ee88cf555fc8c6a1d75de259e9"}
Sep 10 15:33:19.186973 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:19.186916    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-75d99c484-vb9np" podStartSLOduration=2.443787062 podStartE2EDuration="5.186897401s" totalImagesPullingTime="2.743110339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:19.186344993 +0000 UTC m=+42.800471897" watchObservedRunningTime="2026-09-10 15:33:19.186897401 +0000 UTC m=+42.801024305"
Sep 10 15:33:19.611904 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:19.611861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:19.612096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:19.611942    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:19.612096 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:19.611978    2577 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:33:19.612096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:19.611992    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:19.612096 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:19.612048    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:27.612031035 +0000 UTC m=+51.226157920 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : configmap "kube-rbac-proxy" not found
Sep 10 15:33:19.612096 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:19.612076    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:19.612295 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:19.612127    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:27.612113276 +0000 UTC m=+51.226240175 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:19.612295 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:19.612079    2577 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:33:19.612295 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:19.612156    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls podName:f195d304-9189-4b43-839c-25369d19155f nodeName:}" failed. No retries permitted until 2026-09-10 15:33:27.612149557 +0000 UTC m=+51.226276439 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls") pod "dns-default-dp7tl" (UID: "f195d304-9189-4b43-839c-25369d19155f") : secret "dns-default-metrics-tls" not found
Sep 10 15:33:27.674639 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:27.674598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:27.675121 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:27.674653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:27.675121 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:27.674817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:27.675121 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:27.674907    2577 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:27.675121 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:27.674959    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls podName:38279413-0a92-4188-95cb-675c20eafd20 nodeName:}" failed. No retries permitted until 2026-09-10 15:33:43.674946802 +0000 UTC m=+67.289073684 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls") pod "insights-runtime-extractor-h27xd" (UID: "38279413-0a92-4188-95cb-675c20eafd20") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:33:27.675296 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:27.675175    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/38279413-0a92-4188-95cb-675c20eafd20-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:27.678066 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:27.678039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f195d304-9189-4b43-839c-25369d19155f-metrics-tls\") pod \"dns-default-dp7tl\" (UID: \"f195d304-9189-4b43-839c-25369d19155f\") " pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:27.724563 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:27.724539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:28.018084 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:28.018005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dp7tl"]
Sep 10 15:33:28.192862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:28.192822    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dp7tl" event={"ID":"f195d304-9189-4b43-839c-25369d19155f","Type":"ContainerStarted","Data":"73b052247a95a27261c9d39d9a257ecde8a7505ba5038d4e43b19b521c53f7fd"}
Sep 10 15:33:30.200682 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:30.200639    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dp7tl" event={"ID":"f195d304-9189-4b43-839c-25369d19155f","Type":"ContainerStarted","Data":"e6b9bc7a4001dae18c5d5863b64992cc30aff9790c706318e7c795cfb37c7f24"}
Sep 10 15:33:30.201118 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:30.200740    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dp7tl" event={"ID":"f195d304-9189-4b43-839c-25369d19155f","Type":"ContainerStarted","Data":"6e4134eef50ecfcac049de9ec94c75cdd79881c4d3fb1507f4ce054f4c736e7a"}
Sep 10 15:33:30.201118 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:30.200805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:30.225846 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:30.225790    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-dp7tl" podStartSLOduration=17.431008945 podStartE2EDuration="19.225772458s" totalImagesPullingTime="1.794763513s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:30.224327991 +0000 UTC m=+53.838454894" watchObservedRunningTime="2026-09-10 15:33:30.225772458 +0000 UTC m=+53.839899353"
Sep 10 15:33:38.140048 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.140015    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-dk4jq"
Sep 10 15:33:38.798550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.798518    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"]
Sep 10 15:33:38.801445 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.801426    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.805141 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.805119    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 15:33:38.805325 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.805305    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 15:33:38.805580 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.805562    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 15:33:38.806978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.806956    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 15:33:38.820855 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.820829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"]
Sep 10 15:33:38.823142 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.823126    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.824393 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.824368    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"]
Sep 10 15:33:38.827252 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.827119    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 15:33:38.827345 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.827250    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 15:33:38.828063 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.828044    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 15:33:38.828176 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.828157    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 15:33:38.852783 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852746    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2db78d-e39d-4b0d-ada2-4547381fa34d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.852881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852793    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee29cb8-6298-429b-a81e-877041654599-tmp\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.852881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852811    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cee29cb8-6298-429b-a81e-877041654599-klusterlet-config\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.852881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852833    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-ca\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.852881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-hub\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.853042 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852883    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4pzg\" (UniqueName: \"kubernetes.io/projected/cee29cb8-6298-429b-a81e-877041654599-kube-api-access-w4pzg\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.853042 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852902    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmxp2\" (UniqueName: \"kubernetes.io/projected/6f2db78d-e39d-4b0d-ada2-4547381fa34d-kube-api-access-nmxp2\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.853042 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.852970    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.853042 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.853039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.862950 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.862901    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"]
Sep 10 15:33:38.953998 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.953964    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.954170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954027    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.954170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954057    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2db78d-e39d-4b0d-ada2-4547381fa34d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.954170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954076    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee29cb8-6298-429b-a81e-877041654599-tmp\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.954170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954092    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cee29cb8-6298-429b-a81e-877041654599-klusterlet-config\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.954170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954106    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-ca\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.954170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954124    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-hub\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.954461 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954370    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4pzg\" (UniqueName: \"kubernetes.io/projected/cee29cb8-6298-429b-a81e-877041654599-kube-api-access-w4pzg\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.954461 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954420    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmxp2\" (UniqueName: \"kubernetes.io/projected/6f2db78d-e39d-4b0d-ada2-4547381fa34d-kube-api-access-nmxp2\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.954601 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cee29cb8-6298-429b-a81e-877041654599-tmp\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.955037 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.954961    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/6f2db78d-e39d-4b0d-ada2-4547381fa34d-additional-service-ca\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.956798 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.956771    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-hub\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.957036 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.957014    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-ca\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.957139 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.957091    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/cee29cb8-6298-429b-a81e-877041654599-klusterlet-config\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:38.957377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.957358    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.957413 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.957361    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6f2db78d-e39d-4b0d-ada2-4547381fa34d-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.970059 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.970039    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmxp2\" (UniqueName: \"kubernetes.io/projected/6f2db78d-e39d-4b0d-ada2-4547381fa34d-kube-api-access-nmxp2\") pod \"cluster-proxy-proxy-agent-d9cd787f4-gn9c9\" (UID: \"6f2db78d-e39d-4b0d-ada2-4547381fa34d\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:38.970542 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:38.970518    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4pzg\" (UniqueName: \"kubernetes.io/projected/cee29cb8-6298-429b-a81e-877041654599-kube-api-access-w4pzg\") pod \"klusterlet-addon-workmgr-75d8cb4b87-qhq2j\" (UID: \"cee29cb8-6298-429b-a81e-877041654599\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:39.110634 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:39.110596    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:39.140473 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:39.140434    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"
Sep 10 15:33:39.380663 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:39.380594    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"]
Sep 10 15:33:39.383619 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:39.383594    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee29cb8_6298_429b_a81e_877041654599.slice/crio-924b5e5596c38d7097f592aa0011b1713cb2c7f1ca1cce8dceed9bae76431981 WatchSource:0}: Error finding container 924b5e5596c38d7097f592aa0011b1713cb2c7f1ca1cce8dceed9bae76431981: Status 404 returned error can't find the container with id 924b5e5596c38d7097f592aa0011b1713cb2c7f1ca1cce8dceed9bae76431981
Sep 10 15:33:39.419289 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:39.419257    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9"]
Sep 10 15:33:39.422701 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:39.422671    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f2db78d_e39d_4b0d_ada2_4547381fa34d.slice/crio-b023e945504befef6e0ee92744fc7964c66012f8fe0d6c1a348fdaa1f39fe144 WatchSource:0}: Error finding container b023e945504befef6e0ee92744fc7964c66012f8fe0d6c1a348fdaa1f39fe144: Status 404 returned error can't find the container with id b023e945504befef6e0ee92744fc7964c66012f8fe0d6c1a348fdaa1f39fe144
Sep 10 15:33:40.206776 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:40.206543    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dp7tl"
Sep 10 15:33:40.227895 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:40.227858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9" event={"ID":"6f2db78d-e39d-4b0d-ada2-4547381fa34d","Type":"ContainerStarted","Data":"b023e945504befef6e0ee92744fc7964c66012f8fe0d6c1a348fdaa1f39fe144"}
Sep 10 15:33:40.229392 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:40.229357    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j" event={"ID":"cee29cb8-6298-429b-a81e-877041654599","Type":"ContainerStarted","Data":"924b5e5596c38d7097f592aa0011b1713cb2c7f1ca1cce8dceed9bae76431981"}
Sep 10 15:33:42.589538 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.589495    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:42.590030 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.589569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:42.592423 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.592397    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:33:42.593170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.593150    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:33:42.602779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.602588    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:33:42.605157 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.605121    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/08fb1852-cdb8-4abb-97f6-f85fdb8f4651-metrics-certs\") pod \"network-metrics-daemon-qpkv8\" (UID: \"08fb1852-cdb8-4abb-97f6-f85fdb8f4651\") " pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:42.613857 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.613828    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7fkm\" (UniqueName: \"kubernetes.io/projected/6ef725d3-0848-48b2-ae9a-85a288104caa-kube-api-access-m7fkm\") pod \"network-check-target-dx6np\" (UID: \"6ef725d3-0848-48b2-ae9a-85a288104caa\") " pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:42.690940 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.690894    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:42.694139 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.694117    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:33:42.704363 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.704334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/5d4644fb-55db-4320-9d71-ca7b885427f5-original-pull-secret\") pod \"global-pull-secret-syncer-j5nr8\" (UID: \"5d4644fb-55db-4320-9d71-ca7b885427f5\") " pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:42.814202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.814174    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gcptc\""
Sep 10 15:33:42.819929 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.819910    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-skj7z\""
Sep 10 15:33:42.822561 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.822547    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpkv8"
Sep 10 15:33:42.828800 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.828780    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:42.911771 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:42.911733    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-j5nr8"
Sep 10 15:33:43.699019 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:43.698978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:43.701686 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:43.701659    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/38279413-0a92-4188-95cb-675c20eafd20-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-h27xd\" (UID: \"38279413-0a92-4188-95cb-675c20eafd20\") " pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:43.906769 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:43.906722    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-cj5pp\""
Sep 10 15:33:43.914355 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:43.914332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-h27xd"
Sep 10 15:33:45.464737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:45.464644    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dx6np"]
Sep 10 15:33:45.469068 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:45.469039    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef725d3_0848_48b2_ae9a_85a288104caa.slice/crio-4401c3b28892937f7bb0b4fdb04878307b71491cefcebb2f75787fe01064bc47 WatchSource:0}: Error finding container 4401c3b28892937f7bb0b4fdb04878307b71491cefcebb2f75787fe01064bc47: Status 404 returned error can't find the container with id 4401c3b28892937f7bb0b4fdb04878307b71491cefcebb2f75787fe01064bc47
Sep 10 15:33:45.477316 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:45.477292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-j5nr8"]
Sep 10 15:33:45.480111 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:45.480088    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d4644fb_55db_4320_9d71_ca7b885427f5.slice/crio-455e83a1134ca9de56de5e83b87f9b1e5d9640b546d1d950109688ee6d6744f7 WatchSource:0}: Error finding container 455e83a1134ca9de56de5e83b87f9b1e5d9640b546d1d950109688ee6d6744f7: Status 404 returned error can't find the container with id 455e83a1134ca9de56de5e83b87f9b1e5d9640b546d1d950109688ee6d6744f7
Sep 10 15:33:45.481515 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:45.481491    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-h27xd"]
Sep 10 15:33:45.485718 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:45.485694    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38279413_0a92_4188_95cb_675c20eafd20.slice/crio-bbd9c133a5599c5ac7dfe8c25ed5f89e0b5be3460c543ab2017eacf95cee9f03 WatchSource:0}: Error finding container bbd9c133a5599c5ac7dfe8c25ed5f89e0b5be3460c543ab2017eacf95cee9f03: Status 404 returned error can't find the container with id bbd9c133a5599c5ac7dfe8c25ed5f89e0b5be3460c543ab2017eacf95cee9f03
Sep 10 15:33:45.491079 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:45.491060    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qpkv8"]
Sep 10 15:33:45.495366 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:45.495345    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08fb1852_cdb8_4abb_97f6_f85fdb8f4651.slice/crio-32a90713174773f6a1d31bff8608909bc3f9389a290ab014fa71ef186bf8035a WatchSource:0}: Error finding container 32a90713174773f6a1d31bff8608909bc3f9389a290ab014fa71ef186bf8035a: Status 404 returned error can't find the container with id 32a90713174773f6a1d31bff8608909bc3f9389a290ab014fa71ef186bf8035a
Sep 10 15:33:46.254658 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.254550    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j5nr8" event={"ID":"5d4644fb-55db-4320-9d71-ca7b885427f5","Type":"ContainerStarted","Data":"455e83a1134ca9de56de5e83b87f9b1e5d9640b546d1d950109688ee6d6744f7"}
Sep 10 15:33:46.259473 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.258544    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j" event={"ID":"cee29cb8-6298-429b-a81e-877041654599","Type":"ContainerStarted","Data":"85aebd7ae7aff15da0b02e847b498e0c7b5abdae887cf1453a0d445636eeb0c6"}
Sep 10 15:33:46.259473 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.259433    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:46.261268 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.261214    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j"
Sep 10 15:33:46.262782 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.262683    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpkv8" event={"ID":"08fb1852-cdb8-4abb-97f6-f85fdb8f4651","Type":"ContainerStarted","Data":"32a90713174773f6a1d31bff8608909bc3f9389a290ab014fa71ef186bf8035a"}
Sep 10 15:33:46.265043 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.265016    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27xd" event={"ID":"38279413-0a92-4188-95cb-675c20eafd20","Type":"ContainerStarted","Data":"9fd15e258cdd40b8f3f65619a63966c07c5ab6b0f8c7c1298f57bed76086fa36"}
Sep 10 15:33:46.265162 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.265080    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27xd" event={"ID":"38279413-0a92-4188-95cb-675c20eafd20","Type":"ContainerStarted","Data":"bbd9c133a5599c5ac7dfe8c25ed5f89e0b5be3460c543ab2017eacf95cee9f03"}
Sep 10 15:33:46.269440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.269405    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9" event={"ID":"6f2db78d-e39d-4b0d-ada2-4547381fa34d","Type":"ContainerStarted","Data":"9a49050ae454b10f9a79571972bf2d8b808661eebfd989b271b32f7410d6c1c9"}
Sep 10 15:33:46.269440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.269438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9" event={"ID":"6f2db78d-e39d-4b0d-ada2-4547381fa34d","Type":"ContainerStarted","Data":"e50cada0eae64e703e30bda7b8ea92e0364d60ccf6d2390410b4aa1360abf837"}
Sep 10 15:33:46.269598 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.269456    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9" event={"ID":"6f2db78d-e39d-4b0d-ada2-4547381fa34d","Type":"ContainerStarted","Data":"c615f98a38b901177c2d41725246952b3e324184166588d54de8409e03894273"}
Sep 10 15:33:46.271619 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.271575    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dx6np" event={"ID":"6ef725d3-0848-48b2-ae9a-85a288104caa","Type":"ContainerStarted","Data":"4401c3b28892937f7bb0b4fdb04878307b71491cefcebb2f75787fe01064bc47"}
Sep 10 15:33:46.302613 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.302523    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-d9cd787f4-gn9c9" podStartSLOduration=2.531243799 podStartE2EDuration="8.30250293s" totalImagesPullingTime="5.771259131s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:46.301723073 +0000 UTC m=+69.915849977" watchObservedRunningTime="2026-09-10 15:33:46.30250293 +0000 UTC m=+69.916629837"
Sep 10 15:33:46.303215 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:46.303181    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-75d8cb4b87-qhq2j" podStartSLOduration=2.477960904 podStartE2EDuration="8.303170296s" totalImagesPullingTime="5.825209392s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:46.284730997 +0000 UTC m=+69.898857902" watchObservedRunningTime="2026-09-10 15:33:46.303170296 +0000 UTC m=+69.917297204"
Sep 10 15:33:47.279934 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:47.279896    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpkv8" event={"ID":"08fb1852-cdb8-4abb-97f6-f85fdb8f4651","Type":"ContainerStarted","Data":"e82f8db7173f831cfbe9c1d381160e45635092be673f82a1335133121e7a4a2e"}
Sep 10 15:33:47.280370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:47.279962    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpkv8" event={"ID":"08fb1852-cdb8-4abb-97f6-f85fdb8f4651","Type":"ContainerStarted","Data":"551fa7d39462f896493e4252b69d3b73b4691eb9a733bf0f6a634aa1ddc79a49"}
Sep 10 15:33:47.282657 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:47.282619    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27xd" event={"ID":"38279413-0a92-4188-95cb-675c20eafd20","Type":"ContainerStarted","Data":"b9d71ce18e58e68d14478c69137afa26d1a7541f06bea6bad51ba7f78b7abaf4"}
Sep 10 15:33:47.298109 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:47.297581    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qpkv8" podStartSLOduration=68.167543833 podStartE2EDuration="1m9.297559566s" totalImagesPullingTime="1.130015733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:47.295795246 +0000 UTC m=+70.909922166" watchObservedRunningTime="2026-09-10 15:33:47.297559566 +0000 UTC m=+70.911686473"
Sep 10 15:33:49.521308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.520679    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-w4h8q"]
Sep 10 15:33:49.544603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.544575    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.548476 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.548079    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 15:33:49.548476 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.548145    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nmcbx\""
Sep 10 15:33:49.548476 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.548342    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 15:33:49.549720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.549698    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 15:33:49.550479 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.549933    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 15:33:49.550581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.549975    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 15:33:49.550581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.550065    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 15:33:49.650574 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650536    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-tls\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.650574 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650587    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-sys\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.650859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650657    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.650859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-root\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.650859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650745    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7541db6-c688-4047-b38f-3d16ddfc476c-metrics-client-ca\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.650859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650786    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-wtmp\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.650859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650839    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-textfile\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.651114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650867    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.651114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.650892    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nb2gd\" (UniqueName: \"kubernetes.io/projected/d7541db6-c688-4047-b38f-3d16ddfc476c-kube-api-access-nb2gd\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752356    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7541db6-c688-4047-b38f-3d16ddfc476c-metrics-client-ca\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752409    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-wtmp\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-textfile\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752476    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752500    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nb2gd\" (UniqueName: \"kubernetes.io/projected/d7541db6-c688-4047-b38f-3d16ddfc476c-kube-api-access-nb2gd\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-tls\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752569    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-sys\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752595    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752627    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-root\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.752716    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-root\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.753305    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d7541db6-c688-4047-b38f-3d16ddfc476c-metrics-client-ca\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.753434    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-wtmp\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.753992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.753636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-textfile\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.755917 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:49.754665    2577 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 15:33:49.755917 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:33:49.754739    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-tls podName:d7541db6-c688-4047-b38f-3d16ddfc476c nodeName:}" failed. No retries permitted until 2026-09-10 15:33:50.254719964 +0000 UTC m=+73.868846854 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-tls") pod "node-exporter-w4h8q" (UID: "d7541db6-c688-4047-b38f-3d16ddfc476c") : secret "node-exporter-tls" not found
Sep 10 15:33:49.755917 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.755190    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d7541db6-c688-4047-b38f-3d16ddfc476c-sys\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.755917 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.755614    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-accelerators-collector-config\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.761489 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.761440    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:49.765424 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:49.765378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nb2gd\" (UniqueName: \"kubernetes.io/projected/d7541db6-c688-4047-b38f-3d16ddfc476c-kube-api-access-nb2gd\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:50.257510 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:50.257466    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-tls\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:50.260228 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:50.260196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d7541db6-c688-4047-b38f-3d16ddfc476c-node-exporter-tls\") pod \"node-exporter-w4h8q\" (UID: \"d7541db6-c688-4047-b38f-3d16ddfc476c\") " pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:50.456397 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:50.456156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-w4h8q"
Sep 10 15:33:50.756365 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:50.756250    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7541db6_c688_4047_b38f_3d16ddfc476c.slice/crio-b4cfde326ff167db67e3ad3395d8a876d2c621fcdba6f75f11015d41cc81c0cc WatchSource:0}: Error finding container b4cfde326ff167db67e3ad3395d8a876d2c621fcdba6f75f11015d41cc81c0cc: Status 404 returned error can't find the container with id b4cfde326ff167db67e3ad3395d8a876d2c621fcdba6f75f11015d41cc81c0cc
Sep 10 15:33:51.301021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.300981    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dx6np" event={"ID":"6ef725d3-0848-48b2-ae9a-85a288104caa","Type":"ContainerStarted","Data":"fb71fcb7b10e80d3c5adae493ebd7e591db2e03c9f88645eaabe384a25b7b883"}
Sep 10 15:33:51.301224 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.301094    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:33:51.302814 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.302788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-j5nr8" event={"ID":"5d4644fb-55db-4320-9d71-ca7b885427f5","Type":"ContainerStarted","Data":"b114b4c4eadecb19888e9cb0259eeca1a201ea8e63c78d83082317d96d2da025"}
Sep 10 15:33:51.304124 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.304103    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4h8q" event={"ID":"d7541db6-c688-4047-b38f-3d16ddfc476c","Type":"ContainerStarted","Data":"b4cfde326ff167db67e3ad3395d8a876d2c621fcdba6f75f11015d41cc81c0cc"}
Sep 10 15:33:51.306490 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.306465    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-h27xd" event={"ID":"38279413-0a92-4188-95cb-675c20eafd20","Type":"ContainerStarted","Data":"374a5b00d793e20ec414ac4d3e9e2a57d6c6f29a664806737f5fe7cdf5231765"}
Sep 10 15:33:51.321842 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.321793    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dx6np" podStartSLOduration=68.039477739 podStartE2EDuration="1m13.321774058s" totalImagesPullingTime="5.282296319s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:51.321298431 +0000 UTC m=+74.935425336" watchObservedRunningTime="2026-09-10 15:33:51.321774058 +0000 UTC m=+74.935900957"
Sep 10 15:33:51.341983 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.341909    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-h27xd" podStartSLOduration=35.214740581 podStartE2EDuration="40.341894665s" totalImagesPullingTime="5.127154084s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:33:11 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:51.340610403 +0000 UTC m=+74.954737305" watchObservedRunningTime="2026-09-10 15:33:51.341894665 +0000 UTC m=+74.956021569"
Sep 10 15:33:51.357727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:51.357677    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-j5nr8" podStartSLOduration=68.069338227 podStartE2EDuration="1m13.357661134s" totalImagesPullingTime="5.288322907s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:32:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:51.356226345 +0000 UTC m=+74.970353241" watchObservedRunningTime="2026-09-10 15:33:51.357661134 +0000 UTC m=+74.971788037"
Sep 10 15:33:52.311564 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:52.311528    2577 generic.go:356] "Generic (PLEG): container finished" podID="d7541db6-c688-4047-b38f-3d16ddfc476c" containerID="e2782821da26add132d0ca91abd8077bd9bbdc206434ec11998ba02acfe27bee" exitCode=0
Sep 10 15:33:52.312021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:52.311640    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4h8q" event={"ID":"d7541db6-c688-4047-b38f-3d16ddfc476c","Type":"ContainerDied","Data":"e2782821da26add132d0ca91abd8077bd9bbdc206434ec11998ba02acfe27bee"}
Sep 10 15:33:53.318018 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.317985    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4h8q" event={"ID":"d7541db6-c688-4047-b38f-3d16ddfc476c","Type":"ContainerStarted","Data":"b23b8010d764242c5bc391d864bf48d0067493987211276e813e19c487ce92f5"}
Sep 10 15:33:53.318423 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.318037    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-w4h8q" event={"ID":"d7541db6-c688-4047-b38f-3d16ddfc476c","Type":"ContainerStarted","Data":"c283b014825a93635f48fda110767ba0c9a2e9e678a0219872e341f4f5dddfc3"}
Sep 10 15:33:53.338062 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.338010    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-w4h8q" podStartSLOduration=3.449485199 podStartE2EDuration="4.337995126s" totalImagesPullingTime="816.327224ms" totalInitContainerRuntime="72.182703ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:33:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:53.336727388 +0000 UTC m=+76.950854318" watchObservedRunningTime="2026-09-10 15:33:53.337995126 +0000 UTC m=+76.952122081"
Sep 10 15:33:53.523140 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.523107    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b9d97794f-tpwrm"]
Sep 10 15:33:53.525173 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.525156    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.527715 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.527686    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 15:33:53.527876 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.527776    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 15:33:53.527978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.527950    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 15:33:53.528056 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.528027    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 15:33:53.528056 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.528051    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jdlj5\""
Sep 10 15:33:53.528234 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.528150    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 15:33:53.528419 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.528403    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 15:33:53.528419 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.528416    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 15:33:53.533469 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.533454    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 15:33:53.537720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.537701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9d97794f-tpwrm"]
Sep 10 15:33:53.582150 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.582117    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-service-ca\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.582308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.582161    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n8rmk\" (UniqueName: \"kubernetes.io/projected/577f9eec-18d7-414f-a346-848b4e9e680a-kube-api-access-n8rmk\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.582308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.582189    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-serving-cert\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.582308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.582222    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-oauth-config\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.582308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.582255    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-console-config\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.582462 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.582321    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-trusted-ca-bundle\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.582462 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.582348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-oauth-serving-cert\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.683492 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.683451    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-serving-cert\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.683492 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.683492    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-oauth-config\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.683727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.683533    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-console-config\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.683727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.683575    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-trusted-ca-bundle\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.683727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.683598    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-oauth-serving-cert\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.683727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.683668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-service-ca\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.683978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.683817    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n8rmk\" (UniqueName: \"kubernetes.io/projected/577f9eec-18d7-414f-a346-848b4e9e680a-kube-api-access-n8rmk\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.684500 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.684468    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-console-config\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.684607 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.684543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-service-ca\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.684607 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.684588    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-trusted-ca-bundle\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.684812 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.684792    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-oauth-serving-cert\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.686223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.686201    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-oauth-config\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.686309 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.686292    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-serving-cert\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.691804 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.691779    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n8rmk\" (UniqueName: \"kubernetes.io/projected/577f9eec-18d7-414f-a346-848b4e9e680a-kube-api-access-n8rmk\") pod \"console-5b9d97794f-tpwrm\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") " pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:53.835129 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:53.835040    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:33:54.118987 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.118910    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b9d97794f-tpwrm"]
Sep 10 15:33:54.122035 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:54.122010    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod577f9eec_18d7_414f_a346_848b4e9e680a.slice/crio-e2991e5638df340322edd0bd08718fc592072abeb3eb06e4bf0e6d21735d99fc WatchSource:0}: Error finding container e2991e5638df340322edd0bd08718fc592072abeb3eb06e4bf0e6d21735d99fc: Status 404 returned error can't find the container with id e2991e5638df340322edd0bd08718fc592072abeb3eb06e4bf0e6d21735d99fc
Sep 10 15:33:54.274219 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.274188    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"]
Sep 10 15:33:54.276886 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.276871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.280009 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.279987    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gzkhc\""
Sep 10 15:33:54.281474 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.281457    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Sep 10 15:33:54.288974 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.288953    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"]
Sep 10 15:33:54.323013 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.322977    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d97794f-tpwrm" event={"ID":"577f9eec-18d7-414f-a346-848b4e9e680a","Type":"ContainerStarted","Data":"e2991e5638df340322edd0bd08718fc592072abeb3eb06e4bf0e6d21735d99fc"}
Sep 10 15:33:54.389887 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.389792    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlttd\" (UniqueName: \"kubernetes.io/projected/823f89ba-9c71-488a-b391-68a70217eea4-kube-api-access-mlttd\") pod \"monitoring-plugin-85bbbbbf4-7vxk5\" (UID: \"823f89ba-9c71-488a-b391-68a70217eea4\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.390065 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.389963    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/823f89ba-9c71-488a-b391-68a70217eea4-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-7vxk5\" (UID: \"823f89ba-9c71-488a-b391-68a70217eea4\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.490899 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.490851    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/823f89ba-9c71-488a-b391-68a70217eea4-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-7vxk5\" (UID: \"823f89ba-9c71-488a-b391-68a70217eea4\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.490899 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.490910    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlttd\" (UniqueName: \"kubernetes.io/projected/823f89ba-9c71-488a-b391-68a70217eea4-kube-api-access-mlttd\") pod \"monitoring-plugin-85bbbbbf4-7vxk5\" (UID: \"823f89ba-9c71-488a-b391-68a70217eea4\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.493555 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.493529    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/823f89ba-9c71-488a-b391-68a70217eea4-monitoring-plugin-cert\") pod \"monitoring-plugin-85bbbbbf4-7vxk5\" (UID: \"823f89ba-9c71-488a-b391-68a70217eea4\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.500537 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.500516    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlttd\" (UniqueName: \"kubernetes.io/projected/823f89ba-9c71-488a-b391-68a70217eea4-kube-api-access-mlttd\") pod \"monitoring-plugin-85bbbbbf4-7vxk5\" (UID: \"823f89ba-9c71-488a-b391-68a70217eea4\") " pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.585525 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.585492    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:54.868094 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:54.868059    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"]
Sep 10 15:33:54.871402 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:33:54.871371    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod823f89ba_9c71_488a_b391_68a70217eea4.slice/crio-1c91f470e0b8b3bf27ab57b359e51c895131a829a3e2bc2c972ee813fd6f91cf WatchSource:0}: Error finding container 1c91f470e0b8b3bf27ab57b359e51c895131a829a3e2bc2c972ee813fd6f91cf: Status 404 returned error can't find the container with id 1c91f470e0b8b3bf27ab57b359e51c895131a829a3e2bc2c972ee813fd6f91cf
Sep 10 15:33:55.328595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:55.328558    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5" event={"ID":"823f89ba-9c71-488a-b391-68a70217eea4","Type":"ContainerStarted","Data":"1c91f470e0b8b3bf27ab57b359e51c895131a829a3e2bc2c972ee813fd6f91cf"}
Sep 10 15:33:57.342520 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:57.342464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5" event={"ID":"823f89ba-9c71-488a-b391-68a70217eea4","Type":"ContainerStarted","Data":"af94d6a175036af27804fe83649bb06e7414911febee7024e5b2c8e9508d721b"}
Sep 10 15:33:57.342961 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:57.342576    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:33:57.344442 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:57.344409    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d97794f-tpwrm" event={"ID":"577f9eec-18d7-414f-a346-848b4e9e680a","Type":"ContainerStarted","Data":"99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9"}
Sep 10 15:33:57.344609 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:57.344588    2577 patch_prober.go:28] interesting pod/monitoring-plugin-85bbbbbf4-7vxk5 container/monitoring-plugin namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.13:9443/health\": dial tcp 10.134.0.13:9443: connect: connection refused" start-of-body=
Sep 10 15:33:57.344676 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:57.344629    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5" podUID="823f89ba-9c71-488a-b391-68a70217eea4" containerName="monitoring-plugin" probeResult="failure" output="Get \"https://10.134.0.13:9443/health\": dial tcp 10.134.0.13:9443: connect: connection refused"
Sep 10 15:33:57.359561 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:57.359516    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5" podStartSLOduration=0.983161039 podStartE2EDuration="3.359501198s" totalImagesPullingTime="2.376340159s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:57.358818781 +0000 UTC m=+80.972945685" watchObservedRunningTime="2026-09-10 15:33:57.359501198 +0000 UTC m=+80.973628102"
Sep 10 15:33:57.381976 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:57.381924    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5b9d97794f-tpwrm" podStartSLOduration=1.255960783 podStartE2EDuration="4.381905226s" totalImagesPullingTime="3.125944443s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:33:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:33:57.380578751 +0000 UTC m=+80.994705669" watchObservedRunningTime="2026-09-10 15:33:57.381905226 +0000 UTC m=+80.996032134"
Sep 10 15:33:58.352138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:33:58.352110    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-85bbbbbf4-7vxk5"
Sep 10 15:34:00.142051 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:00.142023    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9d97794f-tpwrm"]
Sep 10 15:34:01.993849 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:01.993816    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-59bbfbcc85-nd27j"]
Sep 10 15:34:01.996876 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:01.996846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.008938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.008914    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bbfbcc85-nd27j"]
Sep 10 15:34:02.044696 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.044664    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fkjxg\" (UniqueName: \"kubernetes.io/projected/a7ab0859-d147-4cf5-b757-eea66e41547a-kube-api-access-fkjxg\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.044901 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.044713    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-service-ca\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.044901 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.044736    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-serving-cert\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.044901 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.044781    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-oauth-config\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.044901 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.044835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-oauth-serving-cert\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.044901 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.044852    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-console-config\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.044901 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.044888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-trusted-ca-bundle\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146019 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.145974    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-service-ca\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146019 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-serving-cert\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146233 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146046    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-oauth-config\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146233 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-oauth-serving-cert\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146233 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146110    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-console-config\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146263    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-trusted-ca-bundle\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146322    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fkjxg\" (UniqueName: \"kubernetes.io/projected/a7ab0859-d147-4cf5-b757-eea66e41547a-kube-api-access-fkjxg\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146913 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146842    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-oauth-serving-cert\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.146913 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.146884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-service-ca\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.147167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.147060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-console-config\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.147231 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.147187    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-trusted-ca-bundle\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.148670 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.148644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-serving-cert\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.148771 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.148723    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-oauth-config\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.154933 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.154915    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fkjxg\" (UniqueName: \"kubernetes.io/projected/a7ab0859-d147-4cf5-b757-eea66e41547a-kube-api-access-fkjxg\") pod \"console-59bbfbcc85-nd27j\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") " pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.305353 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.305254    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:02.574094 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:02.574073    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-59bbfbcc85-nd27j"]
Sep 10 15:34:02.576357 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:34:02.576325    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7ab0859_d147_4cf5_b757_eea66e41547a.slice/crio-27d42f004333cef48a7c04121c2520e25c39c8ed085c1debc682045940b62372 WatchSource:0}: Error finding container 27d42f004333cef48a7c04121c2520e25c39c8ed085c1debc682045940b62372: Status 404 returned error can't find the container with id 27d42f004333cef48a7c04121c2520e25c39c8ed085c1debc682045940b62372
Sep 10 15:34:03.368362 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:03.368327    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bbfbcc85-nd27j" event={"ID":"a7ab0859-d147-4cf5-b757-eea66e41547a","Type":"ContainerStarted","Data":"78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c"}
Sep 10 15:34:03.368780 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:03.368379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bbfbcc85-nd27j" event={"ID":"a7ab0859-d147-4cf5-b757-eea66e41547a","Type":"ContainerStarted","Data":"27d42f004333cef48a7c04121c2520e25c39c8ed085c1debc682045940b62372"}
Sep 10 15:34:03.386113 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:03.385877    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-59bbfbcc85-nd27j" podStartSLOduration=2.385859436 podStartE2EDuration="2.385859436s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:34:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:34:03.385293375 +0000 UTC m=+86.999420278" watchObservedRunningTime="2026-09-10 15:34:03.385859436 +0000 UTC m=+86.999986340"
Sep 10 15:34:03.836099 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:03.836068    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:34:12.306071 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:12.306027    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:12.306527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:12.306099    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:12.310932 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:12.310912    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:12.403162 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:12.403138    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:34:22.314410 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:22.314377    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dx6np"
Sep 10 15:34:25.161573 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.161525    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5b9d97794f-tpwrm" podUID="577f9eec-18d7-414f-a346-848b4e9e680a" containerName="console" containerID="cri-o://99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9" gracePeriod=15
Sep 10 15:34:25.419046 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.418987    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:34:25.444540 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.444511    2577 generic.go:356] "Generic (PLEG): container finished" podID="577f9eec-18d7-414f-a346-848b4e9e680a" containerID="99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9" exitCode=0
Sep 10 15:34:25.444718 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.444555    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d97794f-tpwrm" event={"ID":"577f9eec-18d7-414f-a346-848b4e9e680a","Type":"ContainerDied","Data":"99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9"}
Sep 10 15:34:25.444718 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.444576    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b9d97794f-tpwrm"
Sep 10 15:34:25.444718 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.444615    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b9d97794f-tpwrm" event={"ID":"577f9eec-18d7-414f-a346-848b4e9e680a","Type":"ContainerDied","Data":"e2991e5638df340322edd0bd08718fc592072abeb3eb06e4bf0e6d21735d99fc"}
Sep 10 15:34:25.444718 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.444636    2577 scope.go:117] "RemoveContainer" containerID="99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9"
Sep 10 15:34:25.453940 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.453922    2577 scope.go:117] "RemoveContainer" containerID="99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9"
Sep 10 15:34:25.454228 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:34:25.454210    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9\": container with ID starting with 99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9 not found: ID does not exist" containerID="99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9"
Sep 10 15:34:25.454281 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.454245    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9"} err="failed to get container status \"99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9\": rpc error: code = NotFound desc = could not find container \"99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9\": container with ID starting with 99e4f2282cc7583afaaacb1879337ba09c1a9dec86dc0e0859a6d927e56b3df9 not found: ID does not exist"
Sep 10 15:34:25.518487 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.518454    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-serving-cert\") pod \"577f9eec-18d7-414f-a346-848b4e9e680a\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") "
Sep 10 15:34:25.518487 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.518493    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-oauth-serving-cert\") pod \"577f9eec-18d7-414f-a346-848b4e9e680a\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") "
Sep 10 15:34:25.518667 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.518541    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n8rmk\" (UniqueName: \"kubernetes.io/projected/577f9eec-18d7-414f-a346-848b4e9e680a-kube-api-access-n8rmk\") pod \"577f9eec-18d7-414f-a346-848b4e9e680a\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") "
Sep 10 15:34:25.518667 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.518586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-console-config\") pod \"577f9eec-18d7-414f-a346-848b4e9e680a\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") "
Sep 10 15:34:25.518788 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.518745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-service-ca\") pod \"577f9eec-18d7-414f-a346-848b4e9e680a\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") "
Sep 10 15:34:25.518854 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.518827    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-oauth-config\") pod \"577f9eec-18d7-414f-a346-848b4e9e680a\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") "
Sep 10 15:34:25.519017 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.518989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-trusted-ca-bundle\") pod \"577f9eec-18d7-414f-a346-848b4e9e680a\" (UID: \"577f9eec-18d7-414f-a346-848b4e9e680a\") "
Sep 10 15:34:25.519134 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.519065    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-console-config" (OuterVolumeSpecName: "console-config") pod "577f9eec-18d7-414f-a346-848b4e9e680a" (UID: "577f9eec-18d7-414f-a346-848b4e9e680a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:34:25.519134 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.519076    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "577f9eec-18d7-414f-a346-848b4e9e680a" (UID: "577f9eec-18d7-414f-a346-848b4e9e680a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:34:25.519261 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.519147    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-service-ca" (OuterVolumeSpecName: "service-ca") pod "577f9eec-18d7-414f-a346-848b4e9e680a" (UID: "577f9eec-18d7-414f-a346-848b4e9e680a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:34:25.519261 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.519248    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-console-config\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:34:25.519365 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.519266    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:34:25.519365 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.519304    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-oauth-serving-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:34:25.519512 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.519487    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "577f9eec-18d7-414f-a346-848b4e9e680a" (UID: "577f9eec-18d7-414f-a346-848b4e9e680a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:34:25.520957 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.520938    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "577f9eec-18d7-414f-a346-848b4e9e680a" (UID: "577f9eec-18d7-414f-a346-848b4e9e680a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:34:25.521443 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.521409    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "577f9eec-18d7-414f-a346-848b4e9e680a" (UID: "577f9eec-18d7-414f-a346-848b4e9e680a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:34:25.521443 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.521418    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/577f9eec-18d7-414f-a346-848b4e9e680a-kube-api-access-n8rmk" (OuterVolumeSpecName: "kube-api-access-n8rmk") pod "577f9eec-18d7-414f-a346-848b4e9e680a" (UID: "577f9eec-18d7-414f-a346-848b4e9e680a"). InnerVolumeSpecName "kube-api-access-n8rmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:34:25.620536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.620498    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-serving-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:34:25.620536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.620527    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n8rmk\" (UniqueName: \"kubernetes.io/projected/577f9eec-18d7-414f-a346-848b4e9e680a-kube-api-access-n8rmk\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:34:25.620536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.620538    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/577f9eec-18d7-414f-a346-848b4e9e680a-console-oauth-config\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:34:25.620536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.620546    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/577f9eec-18d7-414f-a346-848b4e9e680a-trusted-ca-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:34:25.766281 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.766254    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b9d97794f-tpwrm"]
Sep 10 15:34:25.769835 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:25.769811    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b9d97794f-tpwrm"]
Sep 10 15:34:26.906164 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:26.906131    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="577f9eec-18d7-414f-a346-848b4e9e680a" path="/var/lib/kubelet/pods/577f9eec-18d7-414f-a346-848b4e9e680a/volumes"
Sep 10 15:34:40.350707 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:34:40.350679    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zpnr_e5ff4d54-bda4-480e-9f05-1b4b63e8fc08/serve-healthcheck-canary/0.log"
Sep 10 15:35:43.848559 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:35:43.848482    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bbfbcc85-nd27j"]
Sep 10 15:36:08.867817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:08.867732    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-59bbfbcc85-nd27j" podUID="a7ab0859-d147-4cf5-b757-eea66e41547a" containerName="console" containerID="cri-o://78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c" gracePeriod=15
Sep 10 15:36:09.101147 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.101126    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:36:09.241896 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.241799    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-oauth-config\") pod \"a7ab0859-d147-4cf5-b757-eea66e41547a\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") "
Sep 10 15:36:09.241896 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.241872    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-console-config\") pod \"a7ab0859-d147-4cf5-b757-eea66e41547a\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") "
Sep 10 15:36:09.242115 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.241917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fkjxg\" (UniqueName: \"kubernetes.io/projected/a7ab0859-d147-4cf5-b757-eea66e41547a-kube-api-access-fkjxg\") pod \"a7ab0859-d147-4cf5-b757-eea66e41547a\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") "
Sep 10 15:36:09.242115 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.241933    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-trusted-ca-bundle\") pod \"a7ab0859-d147-4cf5-b757-eea66e41547a\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") "
Sep 10 15:36:09.242115 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.241970    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-service-ca\") pod \"a7ab0859-d147-4cf5-b757-eea66e41547a\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") "
Sep 10 15:36:09.242115 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.242101    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-serving-cert\") pod \"a7ab0859-d147-4cf5-b757-eea66e41547a\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") "
Sep 10 15:36:09.242314 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.242161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-oauth-serving-cert\") pod \"a7ab0859-d147-4cf5-b757-eea66e41547a\" (UID: \"a7ab0859-d147-4cf5-b757-eea66e41547a\") "
Sep 10 15:36:09.242365 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.242303    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-console-config" (OuterVolumeSpecName: "console-config") pod "a7ab0859-d147-4cf5-b757-eea66e41547a" (UID: "a7ab0859-d147-4cf5-b757-eea66e41547a"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:36:09.242365 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.242351    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-service-ca" (OuterVolumeSpecName: "service-ca") pod "a7ab0859-d147-4cf5-b757-eea66e41547a" (UID: "a7ab0859-d147-4cf5-b757-eea66e41547a"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:36:09.242469 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.242402    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "a7ab0859-d147-4cf5-b757-eea66e41547a" (UID: "a7ab0859-d147-4cf5-b757-eea66e41547a"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:36:09.242670 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.242645    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a7ab0859-d147-4cf5-b757-eea66e41547a" (UID: "a7ab0859-d147-4cf5-b757-eea66e41547a"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:36:09.244226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.244192    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a7ab0859-d147-4cf5-b757-eea66e41547a" (UID: "a7ab0859-d147-4cf5-b757-eea66e41547a"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:36:09.244318 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.244232    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7ab0859-d147-4cf5-b757-eea66e41547a-kube-api-access-fkjxg" (OuterVolumeSpecName: "kube-api-access-fkjxg") pod "a7ab0859-d147-4cf5-b757-eea66e41547a" (UID: "a7ab0859-d147-4cf5-b757-eea66e41547a"). InnerVolumeSpecName "kube-api-access-fkjxg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:36:09.244318 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.244263    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a7ab0859-d147-4cf5-b757-eea66e41547a" (UID: "a7ab0859-d147-4cf5-b757-eea66e41547a"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:36:09.342945 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.342908    2577 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-oauth-serving-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:09.342945 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.342939    2577 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-oauth-config\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:09.342945 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.342949    2577 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-console-config\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:09.343175 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.342957    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fkjxg\" (UniqueName: \"kubernetes.io/projected/a7ab0859-d147-4cf5-b757-eea66e41547a-kube-api-access-fkjxg\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:09.343175 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.342966    2577 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-trusted-ca-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:09.343175 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.342974    2577 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a7ab0859-d147-4cf5-b757-eea66e41547a-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:09.343175 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.342982    2577 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a7ab0859-d147-4cf5-b757-eea66e41547a-console-serving-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:09.795518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.795480    2577 generic.go:356] "Generic (PLEG): container finished" podID="a7ab0859-d147-4cf5-b757-eea66e41547a" containerID="78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c" exitCode=0
Sep 10 15:36:09.795716 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.795528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bbfbcc85-nd27j" event={"ID":"a7ab0859-d147-4cf5-b757-eea66e41547a","Type":"ContainerDied","Data":"78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c"}
Sep 10 15:36:09.795716 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.795550    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-59bbfbcc85-nd27j"
Sep 10 15:36:09.795716 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.795599    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-59bbfbcc85-nd27j" event={"ID":"a7ab0859-d147-4cf5-b757-eea66e41547a","Type":"ContainerDied","Data":"27d42f004333cef48a7c04121c2520e25c39c8ed085c1debc682045940b62372"}
Sep 10 15:36:09.795716 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.795620    2577 scope.go:117] "RemoveContainer" containerID="78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c"
Sep 10 15:36:09.805586 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.805567    2577 scope.go:117] "RemoveContainer" containerID="78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c"
Sep 10 15:36:09.805882 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:36:09.805861    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c\": container with ID starting with 78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c not found: ID does not exist" containerID="78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c"
Sep 10 15:36:09.805954 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.805895    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c"} err="failed to get container status \"78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c\": rpc error: code = NotFound desc = could not find container \"78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c\": container with ID starting with 78a7c77ed73f5c9d0698e6e5d6a3ce99ff6634eb744c72015c38beaebee1cc9c not found: ID does not exist"
Sep 10 15:36:09.817819 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.817786    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-59bbfbcc85-nd27j"]
Sep 10 15:36:09.821442 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:09.821420    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-59bbfbcc85-nd27j"]
Sep 10 15:36:10.906638 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:10.906602    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7ab0859-d147-4cf5-b757-eea66e41547a" path="/var/lib/kubelet/pods/a7ab0859-d147-4cf5-b757-eea66e41547a/volumes"
Sep 10 15:36:11.808844 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.808814    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"]
Sep 10 15:36:11.809065 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.809052    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="577f9eec-18d7-414f-a346-848b4e9e680a" containerName="console"
Sep 10 15:36:11.809107 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.809067    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="577f9eec-18d7-414f-a346-848b4e9e680a" containerName="console"
Sep 10 15:36:11.809107 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.809085    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7ab0859-d147-4cf5-b757-eea66e41547a" containerName="console"
Sep 10 15:36:11.809107 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.809091    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7ab0859-d147-4cf5-b757-eea66e41547a" containerName="console"
Sep 10 15:36:11.809196 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.809128    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="577f9eec-18d7-414f-a346-848b4e9e680a" containerName="console"
Sep 10 15:36:11.809196 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.809134    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7ab0859-d147-4cf5-b757-eea66e41547a" containerName="console"
Sep 10 15:36:11.812080 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.812066    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:11.814397 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.814375    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:36:11.815158 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.815143    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:36:11.815241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.815149    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bnqzx\""
Sep 10 15:36:11.820233 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.820212    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"]
Sep 10 15:36:11.964336 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.964293    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:11.964737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.964345    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2fd\" (UniqueName: \"kubernetes.io/projected/9d0c7235-7565-4bb5-878d-6d0ea6485b15-kube-api-access-dt2fd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:11.964737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:11.964420    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.065703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.065624    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.065703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.065680    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2fd\" (UniqueName: \"kubernetes.io/projected/9d0c7235-7565-4bb5-878d-6d0ea6485b15-kube-api-access-dt2fd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.065703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.065706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.066059 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.066040    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.066128 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.066081    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.074787 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.074746    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2fd\" (UniqueName: \"kubernetes.io/projected/9d0c7235-7565-4bb5-878d-6d0ea6485b15-kube-api-access-dt2fd\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.121301 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.121262    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:12.391737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.391707    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"]
Sep 10 15:36:12.394418 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:36:12.394392    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d0c7235_7565_4bb5_878d_6d0ea6485b15.slice/crio-27ef84aaa200916cb0e6265703b3cb4c04e4078ef90dc25aa700b1f27571d2fe WatchSource:0}: Error finding container 27ef84aaa200916cb0e6265703b3cb4c04e4078ef90dc25aa700b1f27571d2fe: Status 404 returned error can't find the container with id 27ef84aaa200916cb0e6265703b3cb4c04e4078ef90dc25aa700b1f27571d2fe
Sep 10 15:36:12.807966 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:12.807883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5" event={"ID":"9d0c7235-7565-4bb5-878d-6d0ea6485b15","Type":"ContainerStarted","Data":"27ef84aaa200916cb0e6265703b3cb4c04e4078ef90dc25aa700b1f27571d2fe"}
Sep 10 15:36:18.832061 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:18.832021    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerID="2fcd9da15669756b309f74b1fee0a370f2e5a44261aeeb57c460d3ef1b7921cf" exitCode=0
Sep 10 15:36:18.832438 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:18.832086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5" event={"ID":"9d0c7235-7565-4bb5-878d-6d0ea6485b15","Type":"ContainerDied","Data":"2fcd9da15669756b309f74b1fee0a370f2e5a44261aeeb57c460d3ef1b7921cf"}
Sep 10 15:36:21.845555 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:21.845516    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerID="afcd5852dd05409e94fa4870b4a89ebf3a8e06bb67e589abb1f969af7ebcfe31" exitCode=0
Sep 10 15:36:21.846020 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:21.845596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5" event={"ID":"9d0c7235-7565-4bb5-878d-6d0ea6485b15","Type":"ContainerDied","Data":"afcd5852dd05409e94fa4870b4a89ebf3a8e06bb67e589abb1f969af7ebcfe31"}
Sep 10 15:36:28.874255 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:28.874225    2577 generic.go:356] "Generic (PLEG): container finished" podID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerID="9c382d12aca8546c6057807e3aa79960e6d8d0a722c363f35d3b038bcdd9d3fc" exitCode=0
Sep 10 15:36:28.874612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:28.874262    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5" event={"ID":"9d0c7235-7565-4bb5-878d-6d0ea6485b15","Type":"ContainerDied","Data":"9c382d12aca8546c6057807e3aa79960e6d8d0a722c363f35d3b038bcdd9d3fc"}
Sep 10 15:36:29.992469 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:29.992446    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:29.996602 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:29.996585    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-bundle\") pod \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") "
Sep 10 15:36:29.996664 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:29.996611    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-util\") pod \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") "
Sep 10 15:36:29.996707 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:29.996662    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dt2fd\" (UniqueName: \"kubernetes.io/projected/9d0c7235-7565-4bb5-878d-6d0ea6485b15-kube-api-access-dt2fd\") pod \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\" (UID: \"9d0c7235-7565-4bb5-878d-6d0ea6485b15\") "
Sep 10 15:36:29.997220 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:29.997195    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-bundle" (OuterVolumeSpecName: "bundle") pod "9d0c7235-7565-4bb5-878d-6d0ea6485b15" (UID: "9d0c7235-7565-4bb5-878d-6d0ea6485b15"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:29.998875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:29.998838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d0c7235-7565-4bb5-878d-6d0ea6485b15-kube-api-access-dt2fd" (OuterVolumeSpecName: "kube-api-access-dt2fd") pod "9d0c7235-7565-4bb5-878d-6d0ea6485b15" (UID: "9d0c7235-7565-4bb5-878d-6d0ea6485b15"). InnerVolumeSpecName "kube-api-access-dt2fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:36:30.002538 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:30.002514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-util" (OuterVolumeSpecName: "util") pod "9d0c7235-7565-4bb5-878d-6d0ea6485b15" (UID: "9d0c7235-7565-4bb5-878d-6d0ea6485b15"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:30.097373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:30.097347    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:30.097373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:30.097367    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9d0c7235-7565-4bb5-878d-6d0ea6485b15-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:30.097373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:30.097376    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2fd\" (UniqueName: \"kubernetes.io/projected/9d0c7235-7565-4bb5-878d-6d0ea6485b15-kube-api-access-dt2fd\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:30.882580 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:30.882549    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5"
Sep 10 15:36:30.882778 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:30.882549    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540cbnz5" event={"ID":"9d0c7235-7565-4bb5-878d-6d0ea6485b15","Type":"ContainerDied","Data":"27ef84aaa200916cb0e6265703b3cb4c04e4078ef90dc25aa700b1f27571d2fe"}
Sep 10 15:36:30.882778 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:30.882671    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27ef84aaa200916cb0e6265703b3cb4c04e4078ef90dc25aa700b1f27571d2fe"
Sep 10 15:36:39.813920 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.813880    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"]
Sep 10 15:36:39.814360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.814115    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerName="pull"
Sep 10 15:36:39.814360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.814126    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerName="pull"
Sep 10 15:36:39.814360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.814139    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerName="util"
Sep 10 15:36:39.814360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.814144    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerName="util"
Sep 10 15:36:39.814360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.814154    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerName="extract"
Sep 10 15:36:39.814360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.814159    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerName="extract"
Sep 10 15:36:39.814360 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.814194    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9d0c7235-7565-4bb5-878d-6d0ea6485b15" containerName="extract"
Sep 10 15:36:39.817720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.817704    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b8flp"]
Sep 10 15:36:39.817854 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.817836    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.819646 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.819625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:39.821208 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.821191    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:36:39.821313 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.821194    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bnqzx\""
Sep 10 15:36:39.821313 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.821226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:36:39.821982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.821968    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-hcnbp\""
Sep 10 15:36:39.822208 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.822197    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 15:36:39.822536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.822525    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 15:36:39.828720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.828701    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b8flp"]
Sep 10 15:36:39.837834 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.837817    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"]
Sep 10 15:36:39.863921 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.863898    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.864021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.863928    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gbms\" (UniqueName: \"kubernetes.io/projected/e0a11743-101a-4e6a-9c2c-2de337573098-kube-api-access-2gbms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.864021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.863951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.864021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.863973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60151b44-4632-4727-8e5d-1043928c1be9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b8flp\" (UID: \"60151b44-4632-4727-8e5d-1043928c1be9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:39.864127 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.864042    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn2xb\" (UniqueName: \"kubernetes.io/projected/60151b44-4632-4727-8e5d-1043928c1be9-kube-api-access-kn2xb\") pod \"cert-manager-webhook-6486ffd997-b8flp\" (UID: \"60151b44-4632-4727-8e5d-1043928c1be9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:39.964941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.964908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.965067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.964950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60151b44-4632-4727-8e5d-1043928c1be9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b8flp\" (UID: \"60151b44-4632-4727-8e5d-1043928c1be9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:39.965067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.964979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn2xb\" (UniqueName: \"kubernetes.io/projected/60151b44-4632-4727-8e5d-1043928c1be9-kube-api-access-kn2xb\") pod \"cert-manager-webhook-6486ffd997-b8flp\" (UID: \"60151b44-4632-4727-8e5d-1043928c1be9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:39.965067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.965013    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.965067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.965034    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gbms\" (UniqueName: \"kubernetes.io/projected/e0a11743-101a-4e6a-9c2c-2de337573098-kube-api-access-2gbms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.965347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.965327    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.965400 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.965378    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:39.973699 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.973672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/60151b44-4632-4727-8e5d-1043928c1be9-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-b8flp\" (UID: \"60151b44-4632-4727-8e5d-1043928c1be9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:39.974165 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.974145    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn2xb\" (UniqueName: \"kubernetes.io/projected/60151b44-4632-4727-8e5d-1043928c1be9-kube-api-access-kn2xb\") pod \"cert-manager-webhook-6486ffd997-b8flp\" (UID: \"60151b44-4632-4727-8e5d-1043928c1be9\") " pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:39.974356 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:39.974337    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gbms\" (UniqueName: \"kubernetes.io/projected/e0a11743-101a-4e6a-9c2c-2de337573098-kube-api-access-2gbms\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:40.127491 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.127465    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:40.133633 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.133612    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:40.399289 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.399266    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"]
Sep 10 15:36:40.401643 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:36:40.401619    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a11743_101a_4e6a_9c2c_2de337573098.slice/crio-d5dd55f512df764c9a8114eae2600c021c8c08e3c74a4c295a4fdba0fb025934 WatchSource:0}: Error finding container d5dd55f512df764c9a8114eae2600c021c8c08e3c74a4c295a4fdba0fb025934: Status 404 returned error can't find the container with id d5dd55f512df764c9a8114eae2600c021c8c08e3c74a4c295a4fdba0fb025934
Sep 10 15:36:40.412869 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.412844    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-b8flp"]
Sep 10 15:36:40.415073 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:36:40.415054    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60151b44_4632_4727_8e5d_1043928c1be9.slice/crio-f3a2a6245c8f71770c24541901facfcf00bab927b18dfaa00d31c34b7027642b WatchSource:0}: Error finding container f3a2a6245c8f71770c24541901facfcf00bab927b18dfaa00d31c34b7027642b: Status 404 returned error can't find the container with id f3a2a6245c8f71770c24541901facfcf00bab927b18dfaa00d31c34b7027642b
Sep 10 15:36:40.915106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.915078    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp" event={"ID":"60151b44-4632-4727-8e5d-1043928c1be9","Type":"ContainerStarted","Data":"f3a2a6245c8f71770c24541901facfcf00bab927b18dfaa00d31c34b7027642b"}
Sep 10 15:36:40.916561 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.916532    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0a11743-101a-4e6a-9c2c-2de337573098" containerID="903b20e86abfb7fc9c608a62bab19c13f447eccccc70008bbb85bc65d55808fe" exitCode=0
Sep 10 15:36:40.916561 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.916562    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz" event={"ID":"e0a11743-101a-4e6a-9c2c-2de337573098","Type":"ContainerDied","Data":"903b20e86abfb7fc9c608a62bab19c13f447eccccc70008bbb85bc65d55808fe"}
Sep 10 15:36:40.916692 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:40.916589    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz" event={"ID":"e0a11743-101a-4e6a-9c2c-2de337573098","Type":"ContainerStarted","Data":"d5dd55f512df764c9a8114eae2600c021c8c08e3c74a4c295a4fdba0fb025934"}
Sep 10 15:36:43.931477 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:43.931435    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0a11743-101a-4e6a-9c2c-2de337573098" containerID="b5538756fb1c7ce34178ec8b29168eecfb11847db7734cf0205735c2020993b8" exitCode=0
Sep 10 15:36:43.931878 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:43.931498    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz" event={"ID":"e0a11743-101a-4e6a-9c2c-2de337573098","Type":"ContainerDied","Data":"b5538756fb1c7ce34178ec8b29168eecfb11847db7734cf0205735c2020993b8"}
Sep 10 15:36:43.933494 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:43.933458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp" event={"ID":"60151b44-4632-4727-8e5d-1043928c1be9","Type":"ContainerStarted","Data":"2794b5b8ec3586f3ef4be5133682c7852bc2a3ef8260abc5a8c0a31a7d82add8"}
Sep 10 15:36:43.933644 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:43.933604    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:43.966907 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:43.966832    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp" podStartSLOduration=1.6814240599999999 podStartE2EDuration="4.966818415s" totalImagesPullingTime="3.285394355s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:36:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:36:43.965971727 +0000 UTC m=+247.580098630" watchObservedRunningTime="2026-09-10 15:36:43.966818415 +0000 UTC m=+247.580945319"
Sep 10 15:36:44.938580 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:44.938549    2577 generic.go:356] "Generic (PLEG): container finished" podID="e0a11743-101a-4e6a-9c2c-2de337573098" containerID="bdd0a41c82392e371b1f29dd45837d5caffed81526632ad8f9a04bb325e9dffc" exitCode=0
Sep 10 15:36:44.939025 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:44.938611    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz" event={"ID":"e0a11743-101a-4e6a-9c2c-2de337573098","Type":"ContainerDied","Data":"bdd0a41c82392e371b1f29dd45837d5caffed81526632ad8f9a04bb325e9dffc"}
Sep 10 15:36:46.058822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.058790    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:46.107926 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.107898    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-util\") pod \"e0a11743-101a-4e6a-9c2c-2de337573098\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") "
Sep 10 15:36:46.108067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.107934    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2gbms\" (UniqueName: \"kubernetes.io/projected/e0a11743-101a-4e6a-9c2c-2de337573098-kube-api-access-2gbms\") pod \"e0a11743-101a-4e6a-9c2c-2de337573098\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") "
Sep 10 15:36:46.108067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.107989    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-bundle\") pod \"e0a11743-101a-4e6a-9c2c-2de337573098\" (UID: \"e0a11743-101a-4e6a-9c2c-2de337573098\") "
Sep 10 15:36:46.108367 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.108345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-bundle" (OuterVolumeSpecName: "bundle") pod "e0a11743-101a-4e6a-9c2c-2de337573098" (UID: "e0a11743-101a-4e6a-9c2c-2de337573098"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:46.110228 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.110205    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a11743-101a-4e6a-9c2c-2de337573098-kube-api-access-2gbms" (OuterVolumeSpecName: "kube-api-access-2gbms") pod "e0a11743-101a-4e6a-9c2c-2de337573098" (UID: "e0a11743-101a-4e6a-9c2c-2de337573098"). InnerVolumeSpecName "kube-api-access-2gbms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:36:46.112593 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.112570    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-util" (OuterVolumeSpecName: "util") pod "e0a11743-101a-4e6a-9c2c-2de337573098" (UID: "e0a11743-101a-4e6a-9c2c-2de337573098"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:36:46.209226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.209147    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:46.209226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.209174    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gbms\" (UniqueName: \"kubernetes.io/projected/e0a11743-101a-4e6a-9c2c-2de337573098-kube-api-access-2gbms\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:46.209226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.209185    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e0a11743-101a-4e6a-9c2c-2de337573098-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:36:46.947090 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.947065    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz"
Sep 10 15:36:46.947226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.947069    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fx2mdz" event={"ID":"e0a11743-101a-4e6a-9c2c-2de337573098","Type":"ContainerDied","Data":"d5dd55f512df764c9a8114eae2600c021c8c08e3c74a4c295a4fdba0fb025934"}
Sep 10 15:36:46.947226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:46.947171    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5dd55f512df764c9a8114eae2600c021c8c08e3c74a4c295a4fdba0fb025934"
Sep 10 15:36:49.940913 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:49.940889    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-b8flp"
Sep 10 15:36:52.951952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.951920    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"]
Sep 10 15:36:52.952310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.952138    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a11743-101a-4e6a-9c2c-2de337573098" containerName="pull"
Sep 10 15:36:52.952310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.952150    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a11743-101a-4e6a-9c2c-2de337573098" containerName="pull"
Sep 10 15:36:52.952310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.952164    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a11743-101a-4e6a-9c2c-2de337573098" containerName="util"
Sep 10 15:36:52.952310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.952169    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a11743-101a-4e6a-9c2c-2de337573098" containerName="util"
Sep 10 15:36:52.952310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.952176    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0a11743-101a-4e6a-9c2c-2de337573098" containerName="extract"
Sep 10 15:36:52.952310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.952181    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0a11743-101a-4e6a-9c2c-2de337573098" containerName="extract"
Sep 10 15:36:52.952310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.952222    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0a11743-101a-4e6a-9c2c-2de337573098" containerName="extract"
Sep 10 15:36:52.955633 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.955617    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:52.958031 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.958011    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:36:52.958182 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.958048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-n8tc9\""
Sep 10 15:36:52.958890 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.958868    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 15:36:52.968990 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:52.968966    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"]
Sep 10 15:36:53.053393 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.053369    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a243ec47-9a4e-4cfd-9ef5-6e74b818ba52-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n6qpv\" (UID: \"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:53.053489 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.053409    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m899t\" (UniqueName: \"kubernetes.io/projected/a243ec47-9a4e-4cfd-9ef5-6e74b818ba52-kube-api-access-m899t\") pod \"openshift-lws-operator-fd8ccff4c-n6qpv\" (UID: \"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:53.154198 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.154175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a243ec47-9a4e-4cfd-9ef5-6e74b818ba52-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n6qpv\" (UID: \"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:53.154283 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.154204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m899t\" (UniqueName: \"kubernetes.io/projected/a243ec47-9a4e-4cfd-9ef5-6e74b818ba52-kube-api-access-m899t\") pod \"openshift-lws-operator-fd8ccff4c-n6qpv\" (UID: \"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:53.154504 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.154487    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a243ec47-9a4e-4cfd-9ef5-6e74b818ba52-tmp\") pod \"openshift-lws-operator-fd8ccff4c-n6qpv\" (UID: \"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:53.161905 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.161884    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m899t\" (UniqueName: \"kubernetes.io/projected/a243ec47-9a4e-4cfd-9ef5-6e74b818ba52-kube-api-access-m899t\") pod \"openshift-lws-operator-fd8ccff4c-n6qpv\" (UID: \"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:53.264709 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.264653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"
Sep 10 15:36:53.532688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.532665    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv"]
Sep 10 15:36:53.535126 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:36:53.535099    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda243ec47_9a4e_4cfd_9ef5_6e74b818ba52.slice/crio-4ac872af09414943c2d08ea54ea0bcde905153161ce4cbc6160c89f6fcfa5459 WatchSource:0}: Error finding container 4ac872af09414943c2d08ea54ea0bcde905153161ce4cbc6160c89f6fcfa5459: Status 404 returned error can't find the container with id 4ac872af09414943c2d08ea54ea0bcde905153161ce4cbc6160c89f6fcfa5459
Sep 10 15:36:53.970881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:53.970850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv" event={"ID":"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52","Type":"ContainerStarted","Data":"4ac872af09414943c2d08ea54ea0bcde905153161ce4cbc6160c89f6fcfa5459"}
Sep 10 15:36:55.980177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:55.980133    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv" event={"ID":"a243ec47-9a4e-4cfd-9ef5-6e74b818ba52","Type":"ContainerStarted","Data":"05b967d200d17a321d896f2a629279c2c9397dc60494be695ce0e7a5f8df4ef2"}
Sep 10 15:36:56.003967 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:56.003899    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-n6qpv" podStartSLOduration=2.371409887 podStartE2EDuration="4.00388168s" totalImagesPullingTime="1.632471793s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:36:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:36:56.003458183 +0000 UTC m=+259.617585086" watchObservedRunningTime="2026-09-10 15:36:56.00388168 +0000 UTC m=+259.618008589"
Sep 10 15:36:59.426577 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.426494    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"]
Sep 10 15:36:59.429421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.429399    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.431860 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.431842    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:36:59.431860 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.431849    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:36:59.432818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.432799    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bnqzx\""
Sep 10 15:36:59.437956 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.437939    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"]
Sep 10 15:36:59.496978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.496953    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.497068 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.496996    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mbpq\" (UniqueName: \"kubernetes.io/projected/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-kube-api-access-4mbpq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.497068 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.497043    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.597399 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.597373    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mbpq\" (UniqueName: \"kubernetes.io/projected/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-kube-api-access-4mbpq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.597491 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.597416    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.597491 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.597441    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.597734 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.597721    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.597804 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.597776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.606692 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.606663    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mbpq\" (UniqueName: \"kubernetes.io/projected/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-kube-api-access-4mbpq\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:36:59.738858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:36:59.738802    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:37:00.013327 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:00.013248    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"]
Sep 10 15:37:00.016088 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:37:00.016064    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a1317ca_949f_4f69_ae85_27c0f46b7ad3.slice/crio-128f713260a5ee66417184144082a10a0ada39354c073810e5547ee781f311ff WatchSource:0}: Error finding container 128f713260a5ee66417184144082a10a0ada39354c073810e5547ee781f311ff: Status 404 returned error can't find the container with id 128f713260a5ee66417184144082a10a0ada39354c073810e5547ee781f311ff
Sep 10 15:37:00.998462 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:00.998384    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerID="fc16ee888153526c8df3a4365752c265a2bfa16e3bd931d81d22171f96052067" exitCode=0
Sep 10 15:37:00.998462 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:00.998428    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7" event={"ID":"4a1317ca-949f-4f69-ae85-27c0f46b7ad3","Type":"ContainerDied","Data":"fc16ee888153526c8df3a4365752c265a2bfa16e3bd931d81d22171f96052067"}
Sep 10 15:37:00.998897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:00.998464    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7" event={"ID":"4a1317ca-949f-4f69-ae85-27c0f46b7ad3","Type":"ContainerStarted","Data":"128f713260a5ee66417184144082a10a0ada39354c073810e5547ee781f311ff"}
Sep 10 15:37:03.006529 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:03.006499    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerID="ad61a95882ce12a618615ab16117a8adb9e2991de66ca93515944226ce6d36cf" exitCode=0
Sep 10 15:37:03.006922 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:03.006579    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7" event={"ID":"4a1317ca-949f-4f69-ae85-27c0f46b7ad3","Type":"ContainerDied","Data":"ad61a95882ce12a618615ab16117a8adb9e2991de66ca93515944226ce6d36cf"}
Sep 10 15:37:04.012482 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:04.012453    2577 generic.go:356] "Generic (PLEG): container finished" podID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerID="b66f826c1a76b6859d221d6114143f9ad5f0cea20e577283b18b22868705fe60" exitCode=0
Sep 10 15:37:04.012913 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:04.012513    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7" event={"ID":"4a1317ca-949f-4f69-ae85-27c0f46b7ad3","Type":"ContainerDied","Data":"b66f826c1a76b6859d221d6114143f9ad5f0cea20e577283b18b22868705fe60"}
Sep 10 15:37:05.132067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.132046    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:37:05.233184 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.233161    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4mbpq\" (UniqueName: \"kubernetes.io/projected/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-kube-api-access-4mbpq\") pod \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") "
Sep 10 15:37:05.233346 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.233193    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-util\") pod \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") "
Sep 10 15:37:05.233346 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.233236    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-bundle\") pod \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\" (UID: \"4a1317ca-949f-4f69-ae85-27c0f46b7ad3\") "
Sep 10 15:37:05.233989 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.233963    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-bundle" (OuterVolumeSpecName: "bundle") pod "4a1317ca-949f-4f69-ae85-27c0f46b7ad3" (UID: "4a1317ca-949f-4f69-ae85-27c0f46b7ad3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:37:05.235377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.235359    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-kube-api-access-4mbpq" (OuterVolumeSpecName: "kube-api-access-4mbpq") pod "4a1317ca-949f-4f69-ae85-27c0f46b7ad3" (UID: "4a1317ca-949f-4f69-ae85-27c0f46b7ad3"). InnerVolumeSpecName "kube-api-access-4mbpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:37:05.239130 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.239097    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-util" (OuterVolumeSpecName: "util") pod "4a1317ca-949f-4f69-ae85-27c0f46b7ad3" (UID: "4a1317ca-949f-4f69-ae85-27c0f46b7ad3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:37:05.333849 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.333812    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:37:05.333849 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.333847    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mbpq\" (UniqueName: \"kubernetes.io/projected/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-kube-api-access-4mbpq\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:37:05.333980 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:05.333860    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4a1317ca-949f-4f69-ae85-27c0f46b7ad3-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:37:06.025292 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:06.025254    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7"
Sep 10 15:37:06.025292 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:06.025267    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af274f7" event={"ID":"4a1317ca-949f-4f69-ae85-27c0f46b7ad3","Type":"ContainerDied","Data":"128f713260a5ee66417184144082a10a0ada39354c073810e5547ee781f311ff"}
Sep 10 15:37:06.025486 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:06.025352    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="128f713260a5ee66417184144082a10a0ada39354c073810e5547ee781f311ff"
Sep 10 15:37:15.058654 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.058615    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"]
Sep 10 15:37:15.059260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.058949    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerName="pull"
Sep 10 15:37:15.059260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.058967    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerName="pull"
Sep 10 15:37:15.059260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.058980    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerName="util"
Sep 10 15:37:15.059260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.058987    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerName="util"
Sep 10 15:37:15.059260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.059006    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerName="extract"
Sep 10 15:37:15.059260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.059014    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerName="extract"
Sep 10 15:37:15.059260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.059071    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a1317ca-949f-4f69-ae85-27c0f46b7ad3" containerName="extract"
Sep 10 15:37:15.063888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.063868    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.066475 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.066451    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:37:15.066575 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.066508    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:37:15.067258 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.067245    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bnqzx\""
Sep 10 15:37:15.070729 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.070567    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"]
Sep 10 15:37:15.100200 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.100165    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.100326 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.100206    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg58d\" (UniqueName: \"kubernetes.io/projected/89d4120d-2520-4a6c-a377-ee59a1b8854c-kube-api-access-tg58d\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.100326 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.100262    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.201224 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.201198    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.201369 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.201235    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg58d\" (UniqueName: \"kubernetes.io/projected/89d4120d-2520-4a6c-a377-ee59a1b8854c-kube-api-access-tg58d\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.201369 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.201265    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.201622 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.201603    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.201622 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.201615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.210052 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.210022    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg58d\" (UniqueName: \"kubernetes.io/projected/89d4120d-2520-4a6c-a377-ee59a1b8854c-kube-api-access-tg58d\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.373570 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.373544    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:15.643107 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:15.643037    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"]
Sep 10 15:37:15.646421 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:37:15.646392    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89d4120d_2520_4a6c_a377_ee59a1b8854c.slice/crio-7a38a2aa68badedcf8a0f12832ddd40b366bd2a212864a6d10f92cbdb457f34f WatchSource:0}: Error finding container 7a38a2aa68badedcf8a0f12832ddd40b366bd2a212864a6d10f92cbdb457f34f: Status 404 returned error can't find the container with id 7a38a2aa68badedcf8a0f12832ddd40b366bd2a212864a6d10f92cbdb457f34f
Sep 10 15:37:16.063397 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:16.063318    2577 generic.go:356] "Generic (PLEG): container finished" podID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerID="d5daa12e65665f9143f79e3448cdf2997b1ba201643f13ef0271f81991832bdb" exitCode=0
Sep 10 15:37:16.063397 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:16.063387    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm" event={"ID":"89d4120d-2520-4a6c-a377-ee59a1b8854c","Type":"ContainerDied","Data":"d5daa12e65665f9143f79e3448cdf2997b1ba201643f13ef0271f81991832bdb"}
Sep 10 15:37:16.063767 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:16.063424    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm" event={"ID":"89d4120d-2520-4a6c-a377-ee59a1b8854c","Type":"ContainerStarted","Data":"7a38a2aa68badedcf8a0f12832ddd40b366bd2a212864a6d10f92cbdb457f34f"}
Sep 10 15:37:17.070283 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.070199    2577 generic.go:356] "Generic (PLEG): container finished" podID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerID="7049e71f03e2dc2b92ef16331c724660f076bc7cb618be5bc615e045098e1389" exitCode=0
Sep 10 15:37:17.070682 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.070276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm" event={"ID":"89d4120d-2520-4a6c-a377-ee59a1b8854c","Type":"ContainerDied","Data":"7049e71f03e2dc2b92ef16331c724660f076bc7cb618be5bc615e045098e1389"}
Sep 10 15:37:17.277135 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.277109    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"]
Sep 10 15:37:17.279311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.279293    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.282146 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.282126    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 15:37:17.282431 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.282415    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 15:37:17.282627 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.282611    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-m59ll\""
Sep 10 15:37:17.282677 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.282639    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 15:37:17.282677 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.282656    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 15:37:17.296339 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.296306    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"]
Sep 10 15:37:17.315132 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.315112    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.315230 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.315143    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8j2q5\" (UniqueName: \"kubernetes.io/projected/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-kube-api-access-8j2q5\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.315230 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.315175    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.315300 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.315252    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-tmp\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.315300 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.315275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-manifests\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.415579 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.415557    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.415719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.415590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-tmp\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.415719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.415609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-manifests\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.415719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.415633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.415719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.415653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8j2q5\" (UniqueName: \"kubernetes.io/projected/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-kube-api-access-8j2q5\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.415990 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.415970    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-manifests\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.418020 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.417990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-tmp\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.418206 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.418191    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.418251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.418241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-webhook-cert\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.429057 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.429037    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8j2q5\" (UniqueName: \"kubernetes.io/projected/4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a-kube-api-access-8j2q5\") pod \"opendatahub-operator-controller-manager-9f4cd445f-nr9h7\" (UID: \"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a\") " pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.588760 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.588735    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:17.863768 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:17.863610    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"]
Sep 10 15:37:17.866325 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:37:17.866297    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b00bcf8_b8dc_4ce4_94c8_c17c197bad8a.slice/crio-0189f4513f36da8cc62ca5ab59c2a225e91351c7153a5542fd13aeb6af1983d5 WatchSource:0}: Error finding container 0189f4513f36da8cc62ca5ab59c2a225e91351c7153a5542fd13aeb6af1983d5: Status 404 returned error can't find the container with id 0189f4513f36da8cc62ca5ab59c2a225e91351c7153a5542fd13aeb6af1983d5
Sep 10 15:37:18.075681 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:18.075649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7" event={"ID":"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a","Type":"ContainerStarted","Data":"0189f4513f36da8cc62ca5ab59c2a225e91351c7153a5542fd13aeb6af1983d5"}
Sep 10 15:37:18.077819 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:18.077795    2577 generic.go:356] "Generic (PLEG): container finished" podID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerID="dd1075dc8c8bc064085b9d69f3c98d8499a59683ea95530ee2b55fef509dc420" exitCode=0
Sep 10 15:37:18.077952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:18.077861    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm" event={"ID":"89d4120d-2520-4a6c-a377-ee59a1b8854c","Type":"ContainerDied","Data":"dd1075dc8c8bc064085b9d69f3c98d8499a59683ea95530ee2b55fef509dc420"}
Sep 10 15:37:19.226502 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.226456    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:19.328018 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.327979    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-bundle\") pod \"89d4120d-2520-4a6c-a377-ee59a1b8854c\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") "
Sep 10 15:37:19.328174 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.328042    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-util\") pod \"89d4120d-2520-4a6c-a377-ee59a1b8854c\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") "
Sep 10 15:37:19.328174 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.328071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tg58d\" (UniqueName: \"kubernetes.io/projected/89d4120d-2520-4a6c-a377-ee59a1b8854c-kube-api-access-tg58d\") pod \"89d4120d-2520-4a6c-a377-ee59a1b8854c\" (UID: \"89d4120d-2520-4a6c-a377-ee59a1b8854c\") "
Sep 10 15:37:19.328822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.328790    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-bundle" (OuterVolumeSpecName: "bundle") pod "89d4120d-2520-4a6c-a377-ee59a1b8854c" (UID: "89d4120d-2520-4a6c-a377-ee59a1b8854c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:37:19.330837 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.330809    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89d4120d-2520-4a6c-a377-ee59a1b8854c-kube-api-access-tg58d" (OuterVolumeSpecName: "kube-api-access-tg58d") pod "89d4120d-2520-4a6c-a377-ee59a1b8854c" (UID: "89d4120d-2520-4a6c-a377-ee59a1b8854c"). InnerVolumeSpecName "kube-api-access-tg58d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:37:19.336743 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.336615    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-util" (OuterVolumeSpecName: "util") pod "89d4120d-2520-4a6c-a377-ee59a1b8854c" (UID: "89d4120d-2520-4a6c-a377-ee59a1b8854c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:37:19.429372 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.429298    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:37:19.429372 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.429332    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/89d4120d-2520-4a6c-a377-ee59a1b8854c-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:37:19.429372 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:19.429347    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg58d\" (UniqueName: \"kubernetes.io/projected/89d4120d-2520-4a6c-a377-ee59a1b8854c-kube-api-access-tg58d\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:37:20.090881 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:20.090848    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm"
Sep 10 15:37:20.091062 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:20.090877    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b02chwm" event={"ID":"89d4120d-2520-4a6c-a377-ee59a1b8854c","Type":"ContainerDied","Data":"7a38a2aa68badedcf8a0f12832ddd40b366bd2a212864a6d10f92cbdb457f34f"}
Sep 10 15:37:20.091062 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:20.090931    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a38a2aa68badedcf8a0f12832ddd40b366bd2a212864a6d10f92cbdb457f34f"
Sep 10 15:37:22.101537 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:22.101501    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7" event={"ID":"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a","Type":"ContainerStarted","Data":"3432e986d4c7f28e3aa437cc43514b6f7c389cd5017bff21bd4485aaf75761b3"}
Sep 10 15:37:23.107043 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:23.107007    2577 generic.go:356] "Generic (PLEG): container finished" podID="4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a" containerID="3432e986d4c7f28e3aa437cc43514b6f7c389cd5017bff21bd4485aaf75761b3" exitCode=0
Sep 10 15:37:23.107410 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:23.107045    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7" event={"ID":"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a","Type":"ContainerDied","Data":"3432e986d4c7f28e3aa437cc43514b6f7c389cd5017bff21bd4485aaf75761b3"}
Sep 10 15:37:24.113535 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:24.113502    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7" event={"ID":"4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a","Type":"ContainerStarted","Data":"18e7e880ec521002b676ca77d6e0120d8787e4687aa074206ab707556994fa58"}
Sep 10 15:37:24.113929 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:24.113614    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:24.137787 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:24.137726    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7" podStartSLOduration=1.9606383219999999 podStartE2EDuration="7.137712806s" totalImagesPullingTime="4.150520027s" totalInitContainerRuntime="1.026554457s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:24.135702199 +0000 UTC m=+287.749829105" watchObservedRunningTime="2026-09-10 15:37:24.137712806 +0000 UTC m=+287.751839710"
Sep 10 15:37:35.120251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:35.120219    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-9f4cd445f-nr9h7"
Sep 10 15:37:36.799168 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:36.799140    2577 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 15:37:37.455934 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.455903    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"]
Sep 10 15:37:37.458866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.456142    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerName="util"
Sep 10 15:37:37.458866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.456152    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerName="util"
Sep 10 15:37:37.458866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.456159    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerName="extract"
Sep 10 15:37:37.458866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.456164    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerName="extract"
Sep 10 15:37:37.458866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.456173    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerName="pull"
Sep 10 15:37:37.458866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.456178    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerName="pull"
Sep 10 15:37:37.458866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.456217    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="89d4120d-2520-4a6c-a377-ee59a1b8854c" containerName="extract"
Sep 10 15:37:37.462974 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.462944    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.465872 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.465839    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 15:37:37.465872 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.465844    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 15:37:37.466246 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.466228    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 15:37:37.466671 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.466652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 15:37:37.475471 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.475451    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"]
Sep 10 15:37:37.562657 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.562627    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tmp\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.562657 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.562661    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tls-certs\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.562868 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.562692    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4pjl\" (UniqueName: \"kubernetes.io/projected/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-kube-api-access-m4pjl\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.663511 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.663477    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4pjl\" (UniqueName: \"kubernetes.io/projected/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-kube-api-access-m4pjl\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.663640 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.663542    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tmp\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.663640 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.663570    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tls-certs\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.663712 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:37:37.663664    2577 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 15:37:37.663742 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:37:37.663736    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tls-certs podName:f5fade1d-b1ca-48ab-a12f-90bdf2c895b0 nodeName:}" failed. No retries permitted until 2026-09-10 15:37:38.163717332 +0000 UTC m=+301.777844213 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tls-certs") pod "kube-auth-proxy-b5b58d48c-cktvx" (UID: "f5fade1d-b1ca-48ab-a12f-90bdf2c895b0") : secret "kube-auth-proxy-tls" not found
Sep 10 15:37:37.666011 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.665994    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tmp\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:37.675645 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:37.675624    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4pjl\" (UniqueName: \"kubernetes.io/projected/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-kube-api-access-m4pjl\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:38.167408 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:38.167374    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tls-certs\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:38.169945 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:38.169917    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f5fade1d-b1ca-48ab-a12f-90bdf2c895b0-tls-certs\") pod \"kube-auth-proxy-b5b58d48c-cktvx\" (UID: \"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0\") " pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:38.372200 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:38.372170    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"
Sep 10 15:37:38.676346 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:38.676267    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx"]
Sep 10 15:37:38.679164 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:37:38.679130    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5fade1d_b1ca_48ab_a12f_90bdf2c895b0.slice/crio-67be0c176cdefa30121af6043be4bd86b519c3bdff3cfc963318a327b5bd87a5 WatchSource:0}: Error finding container 67be0c176cdefa30121af6043be4bd86b519c3bdff3cfc963318a327b5bd87a5: Status 404 returned error can't find the container with id 67be0c176cdefa30121af6043be4bd86b519c3bdff3cfc963318a327b5bd87a5
Sep 10 15:37:38.681039 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:38.681022    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:37:39.078100 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.078069    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"]
Sep 10 15:37:39.082211 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.082196    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.084831 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.084810    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 10 15:37:39.085025 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.085009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 10 15:37:39.085102 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.085009    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dr4sj\""
Sep 10 15:37:39.085102 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.085086    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 15:37:39.085292 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.085275    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 10 15:37:39.096061 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.096038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"]
Sep 10 15:37:39.176238 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.176199    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx" event={"ID":"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0","Type":"ContainerStarted","Data":"67be0c176cdefa30121af6043be4bd86b519c3bdff3cfc963318a327b5bd87a5"}
Sep 10 15:37:39.177473 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.177454    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.177536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.177484    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.177536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.177513    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bb935-a280-49b0-be29-5f75d76f59bd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.177604 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.177583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.177644 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.177612    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.177717 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.177698    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.177797 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.177741    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs8ks\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-kube-api-access-vs8ks\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.278910 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.278876    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.278910 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.278908    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs8ks\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-kube-api-access-vs8ks\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.279143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.278931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.279143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.278948    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.279143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.278986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bb935-a280-49b0-be29-5f75d76f59bd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.279143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.279024    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.279143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.279052    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.279612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.279584    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.281576 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.281553    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bb935-a280-49b0-be29-5f75d76f59bd-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.281657 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.281626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.281832 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.281813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.281885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.281813    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.289536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.289513    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.289713 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.289692    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs8ks\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-kube-api-access-vs8ks\") pod \"istiod-openshift-gateway-7b7995d5cd-6x2zg\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.391720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.391638    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:39.713866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:39.713737    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"]
Sep 10 15:37:39.718675 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:37:39.718627    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod283bb935_a280_49b0_be29_5f75d76f59bd.slice/crio-ac27bf4a97653b943310b1578af6c24ce30862fdd79b007ca9d409ea97088ee4 WatchSource:0}: Error finding container ac27bf4a97653b943310b1578af6c24ce30862fdd79b007ca9d409ea97088ee4: Status 404 returned error can't find the container with id ac27bf4a97653b943310b1578af6c24ce30862fdd79b007ca9d409ea97088ee4
Sep 10 15:37:40.182279 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:40.182241    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg" event={"ID":"283bb935-a280-49b0-be29-5f75d76f59bd","Type":"ContainerStarted","Data":"ac27bf4a97653b943310b1578af6c24ce30862fdd79b007ca9d409ea97088ee4"}
Sep 10 15:37:42.193973 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:42.193928    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx" event={"ID":"f5fade1d-b1ca-48ab-a12f-90bdf2c895b0","Type":"ContainerStarted","Data":"2313281ad792694ac535d4381a959ce5cd608500bafb198e4c92516b23ad86ff"}
Sep 10 15:37:42.212932 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:42.212875    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b5b58d48c-cktvx" podStartSLOduration=2.304540815 podStartE2EDuration="5.212853659s" totalImagesPullingTime="2.908312844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:42.211341174 +0000 UTC m=+305.825468079" watchObservedRunningTime="2026-09-10 15:37:42.212853659 +0000 UTC m=+305.826980565"
Sep 10 15:37:43.365213 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:43.365165    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:37:43.365469 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:43.365243    2577 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892136Ki","pods":"250"}
Sep 10 15:37:44.205046 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:44.205005    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg" event={"ID":"283bb935-a280-49b0-be29-5f75d76f59bd","Type":"ContainerStarted","Data":"50ca8b5fb18c3bc2cb55ef998025248220e8411a0fc281f0d85fbc9ffa18b838"}
Sep 10 15:37:44.205311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:44.205252    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:44.207319 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:44.207290    2577 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-6x2zg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 15:37:44.207443 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:44.207358    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg" podUID="283bb935-a280-49b0-be29-5f75d76f59bd" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:37:44.226386 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:44.226329    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg" podStartSLOduration=1.5822543919999998 podStartE2EDuration="5.226314008s" totalImagesPullingTime="3.644059616s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:37:44.22440556 +0000 UTC m=+307.838532465" watchObservedRunningTime="2026-09-10 15:37:44.226314008 +0000 UTC m=+307.840440912"
Sep 10 15:37:45.209624 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:45.209599    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:37:58.877241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.877204    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"]
Sep 10 15:37:58.880389 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.880371    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:58.888204 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.888181    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Sep 10 15:37:58.888531 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.888515    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-prtt7\""
Sep 10 15:37:58.906709 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.906684    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"]
Sep 10 15:37:58.923208 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.923180    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"]
Sep 10 15:37:58.927128 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.927104    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:58.930948 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.930904    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 15:37:58.931067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.930998    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-92kpd\""
Sep 10 15:37:58.953793 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:58.953764    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"]
Sep 10 15:37:59.025722 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.025678    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xslf\" (UniqueName: \"kubernetes.io/projected/bc8f5d61-999a-46ab-9613-da4dd8616031-kube-api-access-6xslf\") pod \"kserve-controller-manager-d6f4b65b8-59xd9\" (UID: \"bc8f5d61-999a-46ab-9613-da4dd8616031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:59.025929 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.025742    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fxpb\" (UniqueName: \"kubernetes.io/projected/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-kube-api-access-4fxpb\") pod \"llmisvc-controller-manager-777c6bd7d5-gftql\" (UID: \"fea22d52-da84-4c0a-82a9-6f2e7e9e555c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:59.025929 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.025809    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc8f5d61-999a-46ab-9613-da4dd8616031-cert\") pod \"kserve-controller-manager-d6f4b65b8-59xd9\" (UID: \"bc8f5d61-999a-46ab-9613-da4dd8616031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:59.025929 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.025841    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gftql\" (UID: \"fea22d52-da84-4c0a-82a9-6f2e7e9e555c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:59.126986 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.126947    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc8f5d61-999a-46ab-9613-da4dd8616031-cert\") pod \"kserve-controller-manager-d6f4b65b8-59xd9\" (UID: \"bc8f5d61-999a-46ab-9613-da4dd8616031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:59.127188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.127006    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gftql\" (UID: \"fea22d52-da84-4c0a-82a9-6f2e7e9e555c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:59.127188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.127062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xslf\" (UniqueName: \"kubernetes.io/projected/bc8f5d61-999a-46ab-9613-da4dd8616031-kube-api-access-6xslf\") pod \"kserve-controller-manager-d6f4b65b8-59xd9\" (UID: \"bc8f5d61-999a-46ab-9613-da4dd8616031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:59.127188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.127105    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fxpb\" (UniqueName: \"kubernetes.io/projected/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-kube-api-access-4fxpb\") pod \"llmisvc-controller-manager-777c6bd7d5-gftql\" (UID: \"fea22d52-da84-4c0a-82a9-6f2e7e9e555c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:59.127531 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:37:59.127473    2577 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Sep 10 15:37:59.127615 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:37:59.127545    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-cert podName:fea22d52-da84-4c0a-82a9-6f2e7e9e555c nodeName:}" failed. No retries permitted until 2026-09-10 15:37:59.627524786 +0000 UTC m=+323.241651667 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-cert") pod "llmisvc-controller-manager-777c6bd7d5-gftql" (UID: "fea22d52-da84-4c0a-82a9-6f2e7e9e555c") : secret "llmisvc-webhook-server-cert" not found
Sep 10 15:37:59.130421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.130389    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bc8f5d61-999a-46ab-9613-da4dd8616031-cert\") pod \"kserve-controller-manager-d6f4b65b8-59xd9\" (UID: \"bc8f5d61-999a-46ab-9613-da4dd8616031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:59.145879 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.145852    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xslf\" (UniqueName: \"kubernetes.io/projected/bc8f5d61-999a-46ab-9613-da4dd8616031-kube-api-access-6xslf\") pod \"kserve-controller-manager-d6f4b65b8-59xd9\" (UID: \"bc8f5d61-999a-46ab-9613-da4dd8616031\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:59.148691 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.148667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fxpb\" (UniqueName: \"kubernetes.io/projected/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-kube-api-access-4fxpb\") pod \"llmisvc-controller-manager-777c6bd7d5-gftql\" (UID: \"fea22d52-da84-4c0a-82a9-6f2e7e9e555c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:59.191148 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.191103    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:37:59.473823 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.473799    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"]
Sep 10 15:37:59.476212 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:37:59.476190    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc8f5d61_999a_46ab_9613_da4dd8616031.slice/crio-ccae0ca8a938ac382f86915ab5e0c516c23552dcd1113ebe1f77bfab41eddf0a WatchSource:0}: Error finding container ccae0ca8a938ac382f86915ab5e0c516c23552dcd1113ebe1f77bfab41eddf0a: Status 404 returned error can't find the container with id ccae0ca8a938ac382f86915ab5e0c516c23552dcd1113ebe1f77bfab41eddf0a
Sep 10 15:37:59.629946 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.629912    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gftql\" (UID: \"fea22d52-da84-4c0a-82a9-6f2e7e9e555c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:59.632447 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.632427    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fea22d52-da84-4c0a-82a9-6f2e7e9e555c-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gftql\" (UID: \"fea22d52-da84-4c0a-82a9-6f2e7e9e555c\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:37:59.836571 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:37:59.836539    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:38:00.111248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:00.111225    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"]
Sep 10 15:38:00.112895 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:38:00.112869    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podfea22d52_da84_4c0a_82a9_6f2e7e9e555c.slice/crio-c0dd0da1c9a4b9d7a757c43392876aa5bc41ac0ba367a600b9e1ef42451f88d6 WatchSource:0}: Error finding container c0dd0da1c9a4b9d7a757c43392876aa5bc41ac0ba367a600b9e1ef42451f88d6: Status 404 returned error can't find the container with id c0dd0da1c9a4b9d7a757c43392876aa5bc41ac0ba367a600b9e1ef42451f88d6
Sep 10 15:38:00.274020 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:00.273979    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9" event={"ID":"bc8f5d61-999a-46ab-9613-da4dd8616031","Type":"ContainerStarted","Data":"ccae0ca8a938ac382f86915ab5e0c516c23552dcd1113ebe1f77bfab41eddf0a"}
Sep 10 15:38:00.275340 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:00.275313    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql" event={"ID":"fea22d52-da84-4c0a-82a9-6f2e7e9e555c","Type":"ContainerStarted","Data":"c0dd0da1c9a4b9d7a757c43392876aa5bc41ac0ba367a600b9e1ef42451f88d6"}
Sep 10 15:38:00.737612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:00.737579    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"]
Sep 10 15:38:00.737807 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:00.737789    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg" podUID="283bb935-a280-49b0-be29-5f75d76f59bd" containerName="discovery" containerID="cri-o://50ca8b5fb18c3bc2cb55ef998025248220e8411a0fc281f0d85fbc9ffa18b838" gracePeriod=30
Sep 10 15:38:01.282654 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.282616    2577 generic.go:356] "Generic (PLEG): container finished" podID="283bb935-a280-49b0-be29-5f75d76f59bd" containerID="50ca8b5fb18c3bc2cb55ef998025248220e8411a0fc281f0d85fbc9ffa18b838" exitCode=0
Sep 10 15:38:01.283126 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.282677    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg" event={"ID":"283bb935-a280-49b0-be29-5f75d76f59bd","Type":"ContainerDied","Data":"50ca8b5fb18c3bc2cb55ef998025248220e8411a0fc281f0d85fbc9ffa18b838"}
Sep 10 15:38:01.559954 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.559931    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:38:01.647703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.647671    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-cacerts\") pod \"283bb935-a280-49b0-be29-5f75d76f59bd\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") "
Sep 10 15:38:01.647839 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.647729    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs8ks\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-kube-api-access-vs8ks\") pod \"283bb935-a280-49b0-be29-5f75d76f59bd\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") "
Sep 10 15:38:01.647839 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.647796    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-kubeconfig\") pod \"283bb935-a280-49b0-be29-5f75d76f59bd\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") "
Sep 10 15:38:01.647958 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.647842    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-istio-token\") pod \"283bb935-a280-49b0-be29-5f75d76f59bd\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") "
Sep 10 15:38:01.647958 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.647867    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-ca-configmap\") pod \"283bb935-a280-49b0-be29-5f75d76f59bd\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") "
Sep 10 15:38:01.647958 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.647907    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bb935-a280-49b0-be29-5f75d76f59bd-local-certs\") pod \"283bb935-a280-49b0-be29-5f75d76f59bd\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") "
Sep 10 15:38:01.647958 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.647953    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-dns-cert\") pod \"283bb935-a280-49b0-be29-5f75d76f59bd\" (UID: \"283bb935-a280-49b0-be29-5f75d76f59bd\") "
Sep 10 15:38:01.648330 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.648299    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "283bb935-a280-49b0-be29-5f75d76f59bd" (UID: "283bb935-a280-49b0-be29-5f75d76f59bd"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:38:01.650944 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.650882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "283bb935-a280-49b0-be29-5f75d76f59bd" (UID: "283bb935-a280-49b0-be29-5f75d76f59bd"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:38:01.650944 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.650882    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/283bb935-a280-49b0-be29-5f75d76f59bd-local-certs" (OuterVolumeSpecName: "local-certs") pod "283bb935-a280-49b0-be29-5f75d76f59bd" (UID: "283bb935-a280-49b0-be29-5f75d76f59bd"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:01.650944 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.650904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-kube-api-access-vs8ks" (OuterVolumeSpecName: "kube-api-access-vs8ks") pod "283bb935-a280-49b0-be29-5f75d76f59bd" (UID: "283bb935-a280-49b0-be29-5f75d76f59bd"). InnerVolumeSpecName "kube-api-access-vs8ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:38:01.650944 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.650908    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "283bb935-a280-49b0-be29-5f75d76f59bd" (UID: "283bb935-a280-49b0-be29-5f75d76f59bd"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:38:01.650944 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.650921    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-istio-token" (OuterVolumeSpecName: "istio-token") pod "283bb935-a280-49b0-be29-5f75d76f59bd" (UID: "283bb935-a280-49b0-be29-5f75d76f59bd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:38:01.651189 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.651098    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-cacerts" (OuterVolumeSpecName: "cacerts") pod "283bb935-a280-49b0-be29-5f75d76f59bd" (UID: "283bb935-a280-49b0-be29-5f75d76f59bd"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:38:01.748828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.748798    2577 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-cacerts\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:01.748828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.748827    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs8ks\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-kube-api-access-vs8ks\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:01.749009 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.748838    2577 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-kubeconfig\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:01.749009 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.748847    2577 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/283bb935-a280-49b0-be29-5f75d76f59bd-istio-token\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:01.749009 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.748858    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-ca-configmap\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:01.749009 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.748869    2577 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/283bb935-a280-49b0-be29-5f75d76f59bd-local-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:01.749009 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:01.748880    2577 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/283bb935-a280-49b0-be29-5f75d76f59bd-istio-csr-dns-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:02.290483 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:02.290455    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"
Sep 10 15:38:02.290483 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:02.290469    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg" event={"ID":"283bb935-a280-49b0-be29-5f75d76f59bd","Type":"ContainerDied","Data":"ac27bf4a97653b943310b1578af6c24ce30862fdd79b007ca9d409ea97088ee4"}
Sep 10 15:38:02.290948 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:02.290538    2577 scope.go:117] "RemoveContainer" containerID="50ca8b5fb18c3bc2cb55ef998025248220e8411a0fc281f0d85fbc9ffa18b838"
Sep 10 15:38:02.317421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:02.317396    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"]
Sep 10 15:38:02.322768 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:02.322728    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-6x2zg"]
Sep 10 15:38:02.908418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:02.908388    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="283bb935-a280-49b0-be29-5f75d76f59bd" path="/var/lib/kubelet/pods/283bb935-a280-49b0-be29-5f75d76f59bd/volumes"
Sep 10 15:38:03.298801 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:03.298698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9" event={"ID":"bc8f5d61-999a-46ab-9613-da4dd8616031","Type":"ContainerStarted","Data":"6fc71ea6066bbf00f405e4c92876bf7dda9adf598b7488074fd5ba4c4c435204"}
Sep 10 15:38:03.299195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:03.298903    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:38:03.318905 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:03.318846    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9" podStartSLOduration=2.884420598 podStartE2EDuration="5.318823201s" totalImagesPullingTime="2.434402603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:38:03.316710784 +0000 UTC m=+326.930837712" watchObservedRunningTime="2026-09-10 15:38:03.318823201 +0000 UTC m=+326.932950107"
Sep 10 15:38:04.306001 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:04.305960    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql" event={"ID":"fea22d52-da84-4c0a-82a9-6f2e7e9e555c","Type":"ContainerStarted","Data":"f9704c30058f72bf0b911a6ba047952854853c0d1454ac5a161f88712597fb51"}
Sep 10 15:38:04.306385 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:04.306243    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:38:04.339098 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:04.339047    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql" podStartSLOduration=2.980135388 podStartE2EDuration="6.339031415s" totalImagesPullingTime="3.358896027s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:37:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:38:04.337279977 +0000 UTC m=+327.951406899" watchObservedRunningTime="2026-09-10 15:38:04.339031415 +0000 UTC m=+327.953158318"
Sep 10 15:38:34.311724 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:34.311645    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-59xd9"
Sep 10 15:38:35.312143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:35.312115    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gftql"
Sep 10 15:38:53.385063 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.385019    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"]
Sep 10 15:38:53.385530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.385290    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="283bb935-a280-49b0-be29-5f75d76f59bd" containerName="discovery"
Sep 10 15:38:53.385530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.385301    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="283bb935-a280-49b0-be29-5f75d76f59bd" containerName="discovery"
Sep 10 15:38:53.385530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.385355    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="283bb935-a280-49b0-be29-5f75d76f59bd" containerName="discovery"
Sep 10 15:38:53.392058 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.392038    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.394765 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.394727    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 10 15:38:53.395478 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.395452    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-bnqzx\""
Sep 10 15:38:53.395567 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.395455    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 10 15:38:53.402593 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.402574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"]
Sep 10 15:38:53.498735 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.498703    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"]
Sep 10 15:38:53.500548 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.500525    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5mz9\" (UniqueName: \"kubernetes.io/projected/b5cad223-61cb-457d-b88e-70abc78880eb-kube-api-access-x5mz9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.500644 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.500569    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.500644 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.500595    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.501969 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.501952    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.512696 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.512673    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"]
Sep 10 15:38:53.594151 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.594127    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"]
Sep 10 15:38:53.597323 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.597309    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.601918 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.601882    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.602040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.601919    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h8dq\" (UniqueName: \"kubernetes.io/projected/31091418-1df3-459f-a8df-cf8d359bc267-kube-api-access-2h8dq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.602040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.601977    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.602040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.602010    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5mz9\" (UniqueName: \"kubernetes.io/projected/b5cad223-61cb-457d-b88e-70abc78880eb-kube-api-access-x5mz9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.602209 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.602093    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.602209 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.602118    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.602312 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.602295    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-util\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.602370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.602355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-bundle\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.605301 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.605278    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"]
Sep 10 15:38:53.610661 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.610636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5mz9\" (UniqueName: \"kubernetes.io/projected/b5cad223-61cb-457d-b88e-70abc78880eb-kube-api-access-x5mz9\") pod \"ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") " pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.669099 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.669049    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"]
Sep 10 15:38:53.672499 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.672485    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.681620 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.681599    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"]
Sep 10 15:38:53.700519 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.700498    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:53.702819 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.702799    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h8dq\" (UniqueName: \"kubernetes.io/projected/31091418-1df3-459f-a8df-cf8d359bc267-kube-api-access-2h8dq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.702903 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.702841    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.702903 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.702877    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.702903 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.702895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.703059 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.702914    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xd9p9\" (UniqueName: \"kubernetes.io/projected/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-kube-api-access-xd9p9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.703106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.703064    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.703417 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.703291    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-bundle\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.703417 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.703296    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-util\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.711581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.711558    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h8dq\" (UniqueName: \"kubernetes.io/projected/31091418-1df3-459f-a8df-cf8d359bc267-kube-api-access-2h8dq\") pod \"da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") " pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.803617 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.803585    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.803779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.803631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.803779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.803657    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xd9p9\" (UniqueName: \"kubernetes.io/projected/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-kube-api-access-xd9p9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.803779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.803702    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm8r5\" (UniqueName: \"kubernetes.io/projected/4af7bc7e-503a-46dd-9b81-0103cd7fa500-kube-api-access-lm8r5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.803779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.803728    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.803779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.803777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.804044 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.803995    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-bundle\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.804114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.804086    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-util\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.810308 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.810290    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:53.812665 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.812644    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xd9p9\" (UniqueName: \"kubernetes.io/projected/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-kube-api-access-xd9p9\") pod \"6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") " pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.904674 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.904646    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm8r5\" (UniqueName: \"kubernetes.io/projected/4af7bc7e-503a-46dd-9b81-0103cd7fa500-kube-api-access-lm8r5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.904833 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.904681    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.904833 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.904713    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.905064 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.905034    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-bundle\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.905125 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.905072    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-util\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.905554 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.905537    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:53.913569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.913550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm8r5\" (UniqueName: \"kubernetes.io/projected/4af7bc7e-503a-46dd-9b81-0103cd7fa500-kube-api-access-lm8r5\") pod \"e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") " pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:53.971595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.971570    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"]
Sep 10 15:38:53.974126 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:38:53.974097    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5cad223_61cb_457d_b88e_70abc78880eb.slice/crio-1e178ca6b90ac361e5677f001a0ef185537695e91259a7f8c1668eb26bb37e19 WatchSource:0}: Error finding container 1e178ca6b90ac361e5677f001a0ef185537695e91259a7f8c1668eb26bb37e19: Status 404 returned error can't find the container with id 1e178ca6b90ac361e5677f001a0ef185537695e91259a7f8c1668eb26bb37e19
Sep 10 15:38:53.981109 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:53.981087    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:54.081426 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.081402    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"]
Sep 10 15:38:54.083195 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:38:54.083167    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31091418_1df3_459f_a8df_cf8d359bc267.slice/crio-1553e52b37261bc99da16998162af3a8f4d92a5fbfff64cf58f6dca91654a940 WatchSource:0}: Error finding container 1553e52b37261bc99da16998162af3a8f4d92a5fbfff64cf58f6dca91654a940: Status 404 returned error can't find the container with id 1553e52b37261bc99da16998162af3a8f4d92a5fbfff64cf58f6dca91654a940
Sep 10 15:38:54.175617 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.175590    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"]
Sep 10 15:38:54.177281 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:38:54.177257    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc70badd8_37c3_422f_a6b0_a1bc79d6dfcc.slice/crio-7a4bb5c61044f7e30ef848d6419a37264023e8879b9a155cab81ac3a249571dd WatchSource:0}: Error finding container 7a4bb5c61044f7e30ef848d6419a37264023e8879b9a155cab81ac3a249571dd: Status 404 returned error can't find the container with id 7a4bb5c61044f7e30ef848d6419a37264023e8879b9a155cab81ac3a249571dd
Sep 10 15:38:54.252064 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.252041    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"]
Sep 10 15:38:54.274808 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:38:54.274785    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4af7bc7e_503a_46dd_9b81_0103cd7fa500.slice/crio-eb6ca5ffcb2921fd8b0f7aef56c1b3b7f0800518aef2464f837850d75b2aa876 WatchSource:0}: Error finding container eb6ca5ffcb2921fd8b0f7aef56c1b3b7f0800518aef2464f837850d75b2aa876: Status 404 returned error can't find the container with id eb6ca5ffcb2921fd8b0f7aef56c1b3b7f0800518aef2464f837850d75b2aa876
Sep 10 15:38:54.521420 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.521341    2577 generic.go:356] "Generic (PLEG): container finished" podID="31091418-1df3-459f-a8df-cf8d359bc267" containerID="05826fdc454c3a6cbfd34199ced3f01c66552bc6fa2e1fb7166cd3ac4faa1938" exitCode=0
Sep 10 15:38:54.521844 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.521432    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4" event={"ID":"31091418-1df3-459f-a8df-cf8d359bc267","Type":"ContainerDied","Data":"05826fdc454c3a6cbfd34199ced3f01c66552bc6fa2e1fb7166cd3ac4faa1938"}
Sep 10 15:38:54.521844 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.521516    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4" event={"ID":"31091418-1df3-459f-a8df-cf8d359bc267","Type":"ContainerStarted","Data":"1553e52b37261bc99da16998162af3a8f4d92a5fbfff64cf58f6dca91654a940"}
Sep 10 15:38:54.523116 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.523088    2577 generic.go:356] "Generic (PLEG): container finished" podID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerID="3ae932da4cd4395eea0ef5c4d4245c7972adb92207423dad76610c63a9395196" exitCode=0
Sep 10 15:38:54.523241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.523115    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8" event={"ID":"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc","Type":"ContainerDied","Data":"3ae932da4cd4395eea0ef5c4d4245c7972adb92207423dad76610c63a9395196"}
Sep 10 15:38:54.523241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.523156    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8" event={"ID":"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc","Type":"ContainerStarted","Data":"7a4bb5c61044f7e30ef848d6419a37264023e8879b9a155cab81ac3a249571dd"}
Sep 10 15:38:54.524834 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.524814    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5cad223-61cb-457d-b88e-70abc78880eb" containerID="44cf59de7d17f9f6c6f19447e637a83048fffe3bae125566450d27f988b85b0a" exitCode=0
Sep 10 15:38:54.524907 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.524883    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7" event={"ID":"b5cad223-61cb-457d-b88e-70abc78880eb","Type":"ContainerDied","Data":"44cf59de7d17f9f6c6f19447e637a83048fffe3bae125566450d27f988b85b0a"}
Sep 10 15:38:54.524956 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.524930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7" event={"ID":"b5cad223-61cb-457d-b88e-70abc78880eb","Type":"ContainerStarted","Data":"1e178ca6b90ac361e5677f001a0ef185537695e91259a7f8c1668eb26bb37e19"}
Sep 10 15:38:54.526949 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.526925    2577 generic.go:356] "Generic (PLEG): container finished" podID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerID="3b661a3d02276521a0c6f90edb653faf4cbf620f2ed89ae478ee45012d4ca583" exitCode=0
Sep 10 15:38:54.527053 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.526969    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq" event={"ID":"4af7bc7e-503a-46dd-9b81-0103cd7fa500","Type":"ContainerDied","Data":"3b661a3d02276521a0c6f90edb653faf4cbf620f2ed89ae478ee45012d4ca583"}
Sep 10 15:38:54.527053 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:54.526999    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq" event={"ID":"4af7bc7e-503a-46dd-9b81-0103cd7fa500","Type":"ContainerStarted","Data":"eb6ca5ffcb2921fd8b0f7aef56c1b3b7f0800518aef2464f837850d75b2aa876"}
Sep 10 15:38:55.533412 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:55.533337    2577 generic.go:356] "Generic (PLEG): container finished" podID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerID="37eec99fb85b181a271c90935bc19d010e024ca4e97b23d0a28e27ed26bbaa2a" exitCode=0
Sep 10 15:38:55.533743 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:55.533422    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8" event={"ID":"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc","Type":"ContainerDied","Data":"37eec99fb85b181a271c90935bc19d010e024ca4e97b23d0a28e27ed26bbaa2a"}
Sep 10 15:38:56.541379 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.541348    2577 generic.go:356] "Generic (PLEG): container finished" podID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerID="f6debbb4671bce494db96f3afe9cd8aa30fae2e1a54762c89c94aaba271cc0d9" exitCode=0
Sep 10 15:38:56.541784 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.541435    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8" event={"ID":"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc","Type":"ContainerDied","Data":"f6debbb4671bce494db96f3afe9cd8aa30fae2e1a54762c89c94aaba271cc0d9"}
Sep 10 15:38:56.543311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.543278    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5cad223-61cb-457d-b88e-70abc78880eb" containerID="719b07cb84f358c1aae51381484068b1d1d84fe7a33b26c8296e1769e97d841d" exitCode=0
Sep 10 15:38:56.543424 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.543360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7" event={"ID":"b5cad223-61cb-457d-b88e-70abc78880eb","Type":"ContainerDied","Data":"719b07cb84f358c1aae51381484068b1d1d84fe7a33b26c8296e1769e97d841d"}
Sep 10 15:38:56.545297 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.545275    2577 generic.go:356] "Generic (PLEG): container finished" podID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerID="f04aa6ecedd4d3b2747f91bc879460c6d7ee2b377c4e58d22d6ad025a6e102df" exitCode=0
Sep 10 15:38:56.545406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.545361    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq" event={"ID":"4af7bc7e-503a-46dd-9b81-0103cd7fa500","Type":"ContainerDied","Data":"f04aa6ecedd4d3b2747f91bc879460c6d7ee2b377c4e58d22d6ad025a6e102df"}
Sep 10 15:38:56.547334 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.547314    2577 generic.go:356] "Generic (PLEG): container finished" podID="31091418-1df3-459f-a8df-cf8d359bc267" containerID="149b6e984a12c8fdfc91ca5062e944ceee81cdb8315c24badb9cfe81fd360417" exitCode=0
Sep 10 15:38:56.547415 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:56.547360    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4" event={"ID":"31091418-1df3-459f-a8df-cf8d359bc267","Type":"ContainerDied","Data":"149b6e984a12c8fdfc91ca5062e944ceee81cdb8315c24badb9cfe81fd360417"}
Sep 10 15:38:57.554288 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.554255    2577 generic.go:356] "Generic (PLEG): container finished" podID="31091418-1df3-459f-a8df-cf8d359bc267" containerID="a54f567f437b34352e6eec77c81ebff79e4af378d7763bebb5e9930f55534859" exitCode=0
Sep 10 15:38:57.554712 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.554328    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4" event={"ID":"31091418-1df3-459f-a8df-cf8d359bc267","Type":"ContainerDied","Data":"a54f567f437b34352e6eec77c81ebff79e4af378d7763bebb5e9930f55534859"}
Sep 10 15:38:57.556442 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.556421    2577 generic.go:356] "Generic (PLEG): container finished" podID="b5cad223-61cb-457d-b88e-70abc78880eb" containerID="7272e465803e120fd7b168c5226b45b2d8c87b197d392f5fc2df96d47a1ce70c" exitCode=0
Sep 10 15:38:57.556543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.556509    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7" event={"ID":"b5cad223-61cb-457d-b88e-70abc78880eb","Type":"ContainerDied","Data":"7272e465803e120fd7b168c5226b45b2d8c87b197d392f5fc2df96d47a1ce70c"}
Sep 10 15:38:57.558595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.558577    2577 generic.go:356] "Generic (PLEG): container finished" podID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerID="c1f253059d685f1c2a772aece1b9e36b2fedb8bd474c3dbcf8c463968e7bce5f" exitCode=0
Sep 10 15:38:57.558692 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.558638    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq" event={"ID":"4af7bc7e-503a-46dd-9b81-0103cd7fa500","Type":"ContainerDied","Data":"c1f253059d685f1c2a772aece1b9e36b2fedb8bd474c3dbcf8c463968e7bce5f"}
Sep 10 15:38:57.680403 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.680379    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:57.734678 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.734655    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-bundle\") pod \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") "
Sep 10 15:38:57.734822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.734686    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-util\") pod \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") "
Sep 10 15:38:57.734822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.734781    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xd9p9\" (UniqueName: \"kubernetes.io/projected/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-kube-api-access-xd9p9\") pod \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\" (UID: \"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc\") "
Sep 10 15:38:57.735426 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.735362    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-bundle" (OuterVolumeSpecName: "bundle") pod "c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" (UID: "c70badd8-37c3-422f-a6b0-a1bc79d6dfcc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:57.736903 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.736875    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-kube-api-access-xd9p9" (OuterVolumeSpecName: "kube-api-access-xd9p9") pod "c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" (UID: "c70badd8-37c3-422f-a6b0-a1bc79d6dfcc"). InnerVolumeSpecName "kube-api-access-xd9p9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:38:57.739888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.739863    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-util" (OuterVolumeSpecName: "util") pod "c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" (UID: "c70badd8-37c3-422f-a6b0-a1bc79d6dfcc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:57.835482 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.835458    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:57.835482 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.835478    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:57.835608 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:57.835487    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xd9p9\" (UniqueName: \"kubernetes.io/projected/c70badd8-37c3-422f-a6b0-a1bc79d6dfcc-kube-api-access-xd9p9\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.565096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.565070    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8"
Sep 10 15:38:58.565584 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.565113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/6541d93aaee335d6227a8778ab4e840bbe7f5273fe8e69d6bf3410847apcwp8" event={"ID":"c70badd8-37c3-422f-a6b0-a1bc79d6dfcc","Type":"ContainerDied","Data":"7a4bb5c61044f7e30ef848d6419a37264023e8879b9a155cab81ac3a249571dd"}
Sep 10 15:38:58.565584 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.565154    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a4bb5c61044f7e30ef848d6419a37264023e8879b9a155cab81ac3a249571dd"
Sep 10 15:38:58.701996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.701973    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:38:58.742696 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.742673    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:58.745399 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.745382    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:58.842059 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842035    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-util\") pod \"31091418-1df3-459f-a8df-cf8d359bc267\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") "
Sep 10 15:38:58.842203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-bundle\") pod \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") "
Sep 10 15:38:58.842203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842096    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-bundle\") pod \"31091418-1df3-459f-a8df-cf8d359bc267\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") "
Sep 10 15:38:58.842203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842149    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-util\") pod \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") "
Sep 10 15:38:58.842203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842177    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5mz9\" (UniqueName: \"kubernetes.io/projected/b5cad223-61cb-457d-b88e-70abc78880eb-kube-api-access-x5mz9\") pod \"b5cad223-61cb-457d-b88e-70abc78880eb\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") "
Sep 10 15:38:58.842413 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842202    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm8r5\" (UniqueName: \"kubernetes.io/projected/4af7bc7e-503a-46dd-9b81-0103cd7fa500-kube-api-access-lm8r5\") pod \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\" (UID: \"4af7bc7e-503a-46dd-9b81-0103cd7fa500\") "
Sep 10 15:38:58.842413 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842251    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-util\") pod \"b5cad223-61cb-457d-b88e-70abc78880eb\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") "
Sep 10 15:38:58.842413 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842288    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2h8dq\" (UniqueName: \"kubernetes.io/projected/31091418-1df3-459f-a8df-cf8d359bc267-kube-api-access-2h8dq\") pod \"31091418-1df3-459f-a8df-cf8d359bc267\" (UID: \"31091418-1df3-459f-a8df-cf8d359bc267\") "
Sep 10 15:38:58.842413 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842332    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-bundle\") pod \"b5cad223-61cb-457d-b88e-70abc78880eb\" (UID: \"b5cad223-61cb-457d-b88e-70abc78880eb\") "
Sep 10 15:38:58.842947 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842696    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-bundle" (OuterVolumeSpecName: "bundle") pod "31091418-1df3-459f-a8df-cf8d359bc267" (UID: "31091418-1df3-459f-a8df-cf8d359bc267"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:58.842947 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.842904    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-bundle" (OuterVolumeSpecName: "bundle") pod "4af7bc7e-503a-46dd-9b81-0103cd7fa500" (UID: "4af7bc7e-503a-46dd-9b81-0103cd7fa500"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:58.843108 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.843029    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-bundle" (OuterVolumeSpecName: "bundle") pod "b5cad223-61cb-457d-b88e-70abc78880eb" (UID: "b5cad223-61cb-457d-b88e-70abc78880eb"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:58.845183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.845150    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4af7bc7e-503a-46dd-9b81-0103cd7fa500-kube-api-access-lm8r5" (OuterVolumeSpecName: "kube-api-access-lm8r5") pod "4af7bc7e-503a-46dd-9b81-0103cd7fa500" (UID: "4af7bc7e-503a-46dd-9b81-0103cd7fa500"). InnerVolumeSpecName "kube-api-access-lm8r5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:38:58.845285 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.845179    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5cad223-61cb-457d-b88e-70abc78880eb-kube-api-access-x5mz9" (OuterVolumeSpecName: "kube-api-access-x5mz9") pod "b5cad223-61cb-457d-b88e-70abc78880eb" (UID: "b5cad223-61cb-457d-b88e-70abc78880eb"). InnerVolumeSpecName "kube-api-access-x5mz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:38:58.845285 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.845229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31091418-1df3-459f-a8df-cf8d359bc267-kube-api-access-2h8dq" (OuterVolumeSpecName: "kube-api-access-2h8dq") pod "31091418-1df3-459f-a8df-cf8d359bc267" (UID: "31091418-1df3-459f-a8df-cf8d359bc267"). InnerVolumeSpecName "kube-api-access-2h8dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:38:58.848280 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.848252    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-util" (OuterVolumeSpecName: "util") pod "31091418-1df3-459f-a8df-cf8d359bc267" (UID: "31091418-1df3-459f-a8df-cf8d359bc267"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:58.851111 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.851089    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-util" (OuterVolumeSpecName: "util") pod "4af7bc7e-503a-46dd-9b81-0103cd7fa500" (UID: "4af7bc7e-503a-46dd-9b81-0103cd7fa500"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:58.851860 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.851843    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-util" (OuterVolumeSpecName: "util") pod "b5cad223-61cb-457d-b88e-70abc78880eb" (UID: "b5cad223-61cb-457d-b88e-70abc78880eb"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:38:58.943111 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943088    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943111 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943109    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943117    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943124    2577 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/31091418-1df3-459f-a8df-cf8d359bc267-bundle\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943132    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4af7bc7e-503a-46dd-9b81-0103cd7fa500-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943139    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5mz9\" (UniqueName: \"kubernetes.io/projected/b5cad223-61cb-457d-b88e-70abc78880eb-kube-api-access-x5mz9\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943148    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm8r5\" (UniqueName: \"kubernetes.io/projected/4af7bc7e-503a-46dd-9b81-0103cd7fa500-kube-api-access-lm8r5\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943156    2577 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b5cad223-61cb-457d-b88e-70abc78880eb-util\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:58.943239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:58.943163    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h8dq\" (UniqueName: \"kubernetes.io/projected/31091418-1df3-459f-a8df-cf8d359bc267-kube-api-access-2h8dq\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:38:59.571104 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.571068    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4" event={"ID":"31091418-1df3-459f-a8df-cf8d359bc267","Type":"ContainerDied","Data":"1553e52b37261bc99da16998162af3a8f4d92a5fbfff64cf58f6dca91654a940"}
Sep 10 15:38:59.571104 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.571092    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/da112bd609bff87f7bacd2ba941225d7cff15c41059c552dee22e264d7z2pf4"
Sep 10 15:38:59.571607 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.571146    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1553e52b37261bc99da16998162af3a8f4d92a5fbfff64cf58f6dca91654a940"
Sep 10 15:38:59.573092 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.573064    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7" event={"ID":"b5cad223-61cb-457d-b88e-70abc78880eb","Type":"ContainerDied","Data":"1e178ca6b90ac361e5677f001a0ef185537695e91259a7f8c1668eb26bb37e19"}
Sep 10 15:38:59.573223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.573106    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1e178ca6b90ac361e5677f001a0ef185537695e91259a7f8c1668eb26bb37e19"
Sep 10 15:38:59.573223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.573187    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/ea579a99e39e9baa67b0fe2e51b0fa4ef69da3ae0011f15b849816c0c662kz7"
Sep 10 15:38:59.575610 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.575584    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq" event={"ID":"4af7bc7e-503a-46dd-9b81-0103cd7fa500","Type":"ContainerDied","Data":"eb6ca5ffcb2921fd8b0f7aef56c1b3b7f0800518aef2464f837850d75b2aa876"}
Sep 10 15:38:59.575712 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.575624    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb6ca5ffcb2921fd8b0f7aef56c1b3b7f0800518aef2464f837850d75b2aa876"
Sep 10 15:38:59.575712 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:38:59.575635    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e3d011e69d9573cabb8c359e5ac709eb5ac2407647b5169ad26d957e4ct5clq"
Sep 10 15:39:08.309806 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.309774    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5jwrc"]
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310025    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cad223-61cb-457d-b88e-70abc78880eb" containerName="pull"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310036    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cad223-61cb-457d-b88e-70abc78880eb" containerName="pull"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310047    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cad223-61cb-457d-b88e-70abc78880eb" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310054    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cad223-61cb-457d-b88e-70abc78880eb" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310062    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31091418-1df3-459f-a8df-cf8d359bc267" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310067    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31091418-1df3-459f-a8df-cf8d359bc267" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310072    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerName="pull"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310077    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerName="pull"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310082    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310087    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310094    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310098    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310103    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31091418-1df3-459f-a8df-cf8d359bc267" containerName="pull"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310107    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31091418-1df3-459f-a8df-cf8d359bc267" containerName="pull"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310115    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310120    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerName="util"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310126    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310131    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310138    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5cad223-61cb-457d-b88e-70abc78880eb" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310142    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5cad223-61cb-457d-b88e-70abc78880eb" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310147    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31091418-1df3-459f-a8df-cf8d359bc267" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310152    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="31091418-1df3-459f-a8df-cf8d359bc267" containerName="extract"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310161    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerName="pull"
Sep 10 15:39:08.310167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310165    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerName="pull"
Sep 10 15:39:08.310828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310210    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5cad223-61cb-457d-b88e-70abc78880eb" containerName="extract"
Sep 10 15:39:08.310828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310216    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="4af7bc7e-503a-46dd-9b81-0103cd7fa500" containerName="extract"
Sep 10 15:39:08.310828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310223    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="31091418-1df3-459f-a8df-cf8d359bc267" containerName="extract"
Sep 10 15:39:08.310828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.310228    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="c70badd8-37c3-422f-a6b0-a1bc79d6dfcc" containerName="extract"
Sep 10 15:39:08.314275 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.314259    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc"
Sep 10 15:39:08.316922 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.316901    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 15:39:08.317155 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.317140    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-ps82h\""
Sep 10 15:39:08.317620 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.317604    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 15:39:08.323837 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.323816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5jwrc"]
Sep 10 15:39:08.417784 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.417737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gm48t\" (UniqueName: \"kubernetes.io/projected/18e6aad0-75ad-4a8a-9401-ff5378e30609-kube-api-access-gm48t\") pod \"authorino-operator-76b478bd78-5jwrc\" (UID: \"18e6aad0-75ad-4a8a-9401-ff5378e30609\") " pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc"
Sep 10 15:39:08.519085 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.519044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gm48t\" (UniqueName: \"kubernetes.io/projected/18e6aad0-75ad-4a8a-9401-ff5378e30609-kube-api-access-gm48t\") pod \"authorino-operator-76b478bd78-5jwrc\" (UID: \"18e6aad0-75ad-4a8a-9401-ff5378e30609\") " pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc"
Sep 10 15:39:08.529139 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.529113    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gm48t\" (UniqueName: \"kubernetes.io/projected/18e6aad0-75ad-4a8a-9401-ff5378e30609-kube-api-access-gm48t\") pod \"authorino-operator-76b478bd78-5jwrc\" (UID: \"18e6aad0-75ad-4a8a-9401-ff5378e30609\") " pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc"
Sep 10 15:39:08.623566 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.623540    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc"
Sep 10 15:39:08.899949 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:08.899861    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-76b478bd78-5jwrc"]
Sep 10 15:39:08.903268 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:08.903241    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e6aad0_75ad_4a8a_9401_ff5378e30609.slice/crio-4766a0e0b0e6b6fa6cbbd3bfe7ffc92de99af633cdd6cc915ca86ff9f5ba830c WatchSource:0}: Error finding container 4766a0e0b0e6b6fa6cbbd3bfe7ffc92de99af633cdd6cc915ca86ff9f5ba830c: Status 404 returned error can't find the container with id 4766a0e0b0e6b6fa6cbbd3bfe7ffc92de99af633cdd6cc915ca86ff9f5ba830c
Sep 10 15:39:09.620120 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:09.620086    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc" event={"ID":"18e6aad0-75ad-4a8a-9401-ff5378e30609","Type":"ContainerStarted","Data":"4766a0e0b0e6b6fa6cbbd3bfe7ffc92de99af633cdd6cc915ca86ff9f5ba830c"}
Sep 10 15:39:11.369355 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.369315    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"]
Sep 10 15:39:11.373872 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.373846    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"
Sep 10 15:39:11.376461 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.376441    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-848qd\""
Sep 10 15:39:11.376726 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.376708    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 10 15:39:11.389728 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.389705    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"]
Sep 10 15:39:11.543037 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.543000    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pf5t4\" (UniqueName: \"kubernetes.io/projected/c6def56d-78d9-4db4-9209-819e5349fbed-kube-api-access-pf5t4\") pod \"dns-operator-controller-manager-7b84cd7c9c-j42hr\" (UID: \"c6def56d-78d9-4db4-9209-819e5349fbed\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"
Sep 10 15:39:11.643715 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.643620    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pf5t4\" (UniqueName: \"kubernetes.io/projected/c6def56d-78d9-4db4-9209-819e5349fbed-kube-api-access-pf5t4\") pod \"dns-operator-controller-manager-7b84cd7c9c-j42hr\" (UID: \"c6def56d-78d9-4db4-9209-819e5349fbed\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"
Sep 10 15:39:11.651845 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.651819    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pf5t4\" (UniqueName: \"kubernetes.io/projected/c6def56d-78d9-4db4-9209-819e5349fbed-kube-api-access-pf5t4\") pod \"dns-operator-controller-manager-7b84cd7c9c-j42hr\" (UID: \"c6def56d-78d9-4db4-9209-819e5349fbed\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"
Sep 10 15:39:11.683818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.683787    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"
Sep 10 15:39:11.965200 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:11.965177    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"]
Sep 10 15:39:11.967001 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:11.966963    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6def56d_78d9_4db4_9209_819e5349fbed.slice/crio-6883fcc1f9a656d45d7c3ccdde9c3eb0b95213d27a3b00796af6a12948c5e72d WatchSource:0}: Error finding container 6883fcc1f9a656d45d7c3ccdde9c3eb0b95213d27a3b00796af6a12948c5e72d: Status 404 returned error can't find the container with id 6883fcc1f9a656d45d7c3ccdde9c3eb0b95213d27a3b00796af6a12948c5e72d
Sep 10 15:39:12.635878 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:12.635844    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr" event={"ID":"c6def56d-78d9-4db4-9209-819e5349fbed","Type":"ContainerStarted","Data":"6883fcc1f9a656d45d7c3ccdde9c3eb0b95213d27a3b00796af6a12948c5e72d"}
Sep 10 15:39:14.415383 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.415359    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"]
Sep 10 15:39:14.418408 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.418389    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:14.421073 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.421048    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-xmr82\""
Sep 10 15:39:14.433871 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.433848    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"]
Sep 10 15:39:14.564108 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.564066    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bgpz\" (UniqueName: \"kubernetes.io/projected/169a3d34-d81a-4746-a1eb-317dd0a5e8eb-kube-api-access-6bgpz\") pod \"limitador-operator-controller-manager-74495c69f8-qxmrn\" (UID: \"169a3d34-d81a-4746-a1eb-317dd0a5e8eb\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:14.647732 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.647651    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc" event={"ID":"18e6aad0-75ad-4a8a-9401-ff5378e30609","Type":"ContainerStarted","Data":"98814c8175c4c0055ccae36b7d9a82c4f1956a5f30a9b07bbfa43720b8ba5177"}
Sep 10 15:39:14.647883 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.647773    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc"
Sep 10 15:39:14.665114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.665089    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bgpz\" (UniqueName: \"kubernetes.io/projected/169a3d34-d81a-4746-a1eb-317dd0a5e8eb-kube-api-access-6bgpz\") pod \"limitador-operator-controller-manager-74495c69f8-qxmrn\" (UID: \"169a3d34-d81a-4746-a1eb-317dd0a5e8eb\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:14.672293 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.672242    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc" podStartSLOduration=1.217471337 podStartE2EDuration="6.672228501s" totalImagesPullingTime="5.454757164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:14.67023057 +0000 UTC m=+398.284357486" watchObservedRunningTime="2026-09-10 15:39:14.672228501 +0000 UTC m=+398.286355404"
Sep 10 15:39:14.676673 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.676651    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bgpz\" (UniqueName: \"kubernetes.io/projected/169a3d34-d81a-4746-a1eb-317dd0a5e8eb-kube-api-access-6bgpz\") pod \"limitador-operator-controller-manager-74495c69f8-qxmrn\" (UID: \"169a3d34-d81a-4746-a1eb-317dd0a5e8eb\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:14.728027 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:14.727994    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:15.003403 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:15.003338    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"]
Sep 10 15:39:15.008334 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:15.008304    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod169a3d34_d81a_4746_a1eb_317dd0a5e8eb.slice/crio-29652000a92b4177a09662064dcf8543f08f4c2fb161e3f65c08774f39e13f46 WatchSource:0}: Error finding container 29652000a92b4177a09662064dcf8543f08f4c2fb161e3f65c08774f39e13f46: Status 404 returned error can't find the container with id 29652000a92b4177a09662064dcf8543f08f4c2fb161e3f65c08774f39e13f46
Sep 10 15:39:15.653536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:15.653497    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn" event={"ID":"169a3d34-d81a-4746-a1eb-317dd0a5e8eb","Type":"ContainerStarted","Data":"29652000a92b4177a09662064dcf8543f08f4c2fb161e3f65c08774f39e13f46"}
Sep 10 15:39:17.666488 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:17.666452    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn" event={"ID":"169a3d34-d81a-4746-a1eb-317dd0a5e8eb","Type":"ContainerStarted","Data":"780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765"}
Sep 10 15:39:17.666996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:17.666581    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:17.684661 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:17.684617    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn" podStartSLOduration=1.444116401 podStartE2EDuration="3.684604007s" totalImagesPullingTime="2.240487606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:17.683288292 +0000 UTC m=+401.297415197" watchObservedRunningTime="2026-09-10 15:39:17.684604007 +0000 UTC m=+401.298730964"
Sep 10 15:39:20.344076 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.343997    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"]
Sep 10 15:39:20.347385 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.347363    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:20.350031 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.350013    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-865x4\""
Sep 10 15:39:20.362198 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.362173    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"]
Sep 10 15:39:20.516387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.516348    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:20.516561 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.516395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fchjd\" (UniqueName: \"kubernetes.io/projected/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-kube-api-access-fchjd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:20.617763 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.617666    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:20.617763 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.617709    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fchjd\" (UniqueName: \"kubernetes.io/projected/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-kube-api-access-fchjd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:20.618160 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.618125    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:20.628269 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.628241    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fchjd\" (UniqueName: \"kubernetes.io/projected/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-kube-api-access-fchjd\") pod \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:20.656989 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:20.656954    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:21.056235 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:21.056213    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"]
Sep 10 15:39:21.058464 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:21.058434    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c9b2d42_60b3_48df_8f36_ecc395ba6f48.slice/crio-91d9ece8c9eafb2d82db3e4c35cba8f4e7f521a821a515481a77b4cf5d912094 WatchSource:0}: Error finding container 91d9ece8c9eafb2d82db3e4c35cba8f4e7f521a821a515481a77b4cf5d912094: Status 404 returned error can't find the container with id 91d9ece8c9eafb2d82db3e4c35cba8f4e7f521a821a515481a77b4cf5d912094
Sep 10 15:39:21.686909 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:21.686871    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr" event={"ID":"5c9b2d42-60b3-48df-8f36-ecc395ba6f48","Type":"ContainerStarted","Data":"91d9ece8c9eafb2d82db3e4c35cba8f4e7f521a821a515481a77b4cf5d912094"}
Sep 10 15:39:21.688513 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:21.688492    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr" event={"ID":"c6def56d-78d9-4db4-9209-819e5349fbed","Type":"ContainerStarted","Data":"fc1baf22439c20fdc8a91a2ee5c90fc75c6cf17c782261d2011d2a691b415339"}
Sep 10 15:39:21.688636 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:21.688607    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"
Sep 10 15:39:21.708656 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:21.708604    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr" podStartSLOduration=1.8287006730000002 podStartE2EDuration="10.708587837s" totalImagesPullingTime="8.879887164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:21.708288033 +0000 UTC m=+405.322414936" watchObservedRunningTime="2026-09-10 15:39:21.708587837 +0000 UTC m=+405.322714741"
Sep 10 15:39:25.656468 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:25.656436    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-76b478bd78-5jwrc"
Sep 10 15:39:25.713966 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:25.713927    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr" event={"ID":"5c9b2d42-60b3-48df-8f36-ecc395ba6f48","Type":"ContainerStarted","Data":"28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6"}
Sep 10 15:39:25.714165 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:25.714122    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:25.735686 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:25.735627    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr" podStartSLOduration=1.835239986 podStartE2EDuration="5.73561157s" totalImagesPullingTime="3.900371584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:25.733938038 +0000 UTC m=+409.348064946" watchObservedRunningTime="2026-09-10 15:39:25.73561157 +0000 UTC m=+409.349738473"
Sep 10 15:39:28.673746 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:28.673714    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:32.696871 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:32.696825    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-j42hr"
Sep 10 15:39:36.721286 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:36.721254    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:37.857682 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:37.857648    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"]
Sep 10 15:39:37.862250 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:37.862233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:37.873884 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:37.873858    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"]
Sep 10 15:39:37.946118 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:37.946086    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:37.946296 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:37.946133    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f84tp\" (UniqueName: \"kubernetes.io/projected/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-kube-api-access-f84tp\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:38.046992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.046959    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f84tp\" (UniqueName: \"kubernetes.io/projected/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-kube-api-access-f84tp\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:38.047176 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.047035    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:38.047387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.047371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:38.056697 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.056669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f84tp\" (UniqueName: \"kubernetes.io/projected/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-kube-api-access-f84tp\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:38.172765 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.172655    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:38.449000 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.448921    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"]
Sep 10 15:39:38.453038 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:38.453015    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74f77ff1_2ffa_4d94_ae2b_c5751acf6422.slice/crio-7a7bfdd2d2385d7148b3eb5cf68f17ea7b5ebeef401643a0840174f69b96741b WatchSource:0}: Error finding container 7a7bfdd2d2385d7148b3eb5cf68f17ea7b5ebeef401643a0840174f69b96741b: Status 404 returned error can't find the container with id 7a7bfdd2d2385d7148b3eb5cf68f17ea7b5ebeef401643a0840174f69b96741b
Sep 10 15:39:38.779896 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.779798    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr" event={"ID":"74f77ff1-2ffa-4d94-ae2b-c5751acf6422","Type":"ContainerStarted","Data":"672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898"}
Sep 10 15:39:38.779896 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.779850    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr" event={"ID":"74f77ff1-2ffa-4d94-ae2b-c5751acf6422","Type":"ContainerStarted","Data":"7a7bfdd2d2385d7148b3eb5cf68f17ea7b5ebeef401643a0840174f69b96741b"}
Sep 10 15:39:38.779896 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.779879    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:38.801653 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.801604    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr" podStartSLOduration=1.801587923 podStartE2EDuration="1.801587923s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:38.800600727 +0000 UTC m=+422.414727630" watchObservedRunningTime="2026-09-10 15:39:38.801587923 +0000 UTC m=+422.415714826"
Sep 10 15:39:38.927073 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.927043    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"]
Sep 10 15:39:38.931666 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.931639    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"]
Sep 10 15:39:38.947984 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.947957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"]
Sep 10 15:39:38.948160 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.948136    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" containerName="manager" containerID="cri-o://28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6" gracePeriod=2
Sep 10 15:39:38.958856 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.958829    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"]
Sep 10 15:39:38.959246 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.959228    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" containerName="manager"
Sep 10 15:39:38.959373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.959251    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" containerName="manager"
Sep 10 15:39:38.959373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.959351    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" containerName="manager"
Sep 10 15:39:38.963033 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.963013    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:38.963138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.963069    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"]
Sep 10 15:39:38.979527 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.979500    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"]
Sep 10 15:39:38.979791 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.979739    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" containerName="manager" containerID="cri-o://780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765" gracePeriod=2
Sep 10 15:39:38.980605 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:38.980571    2577 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\": pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:38.981735 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.981715    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"]
Sep 10 15:39:38.982152 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.982132    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" containerName="manager"
Sep 10 15:39:38.982232 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.982155    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" containerName="manager"
Sep 10 15:39:38.982297 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.982282    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" containerName="manager"
Sep 10 15:39:38.997105 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.990704    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"]
Sep 10 15:39:38.997105 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.990844    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:38.997105 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.992944    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"]
Sep 10 15:39:38.998006 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:38.997981    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"]
Sep 10 15:39:39.024799 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.024548    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.027780 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.026744    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.033487 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.033429    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"]
Sep 10 15:39:39.033959 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.033908    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" containerName="manager"
Sep 10 15:39:39.033959 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.033929    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" containerName="manager"
Sep 10 15:39:39.034106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.034010    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" containerName="manager"
Sep 10 15:39:39.038117 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.038099    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"
Sep 10 15:39:39.038296 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.038279    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"]
Sep 10 15:39:39.055542 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.055517    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fda158bd-2217-4935-a3e0-32e1f88b009b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-btpqq\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:39.055678 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.055574    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7t24\" (UniqueName: \"kubernetes.io/projected/53501126-73af-4b23-888c-e9d1660de15b-kube-api-access-r7t24\") pod \"limitador-operator-controller-manager-74495c69f8-sppbt\" (UID: \"53501126-73af-4b23-888c-e9d1660de15b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"
Sep 10 15:39:39.055775 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.055682    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kshqn\" (UniqueName: \"kubernetes.io/projected/fda158bd-2217-4935-a3e0-32e1f88b009b-kube-api-access-kshqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-btpqq\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:39.055775 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.055721    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffac071d-02fa-4e7a-9f0b-eed903c73d85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-m89q4\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.056056 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.055794    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2dk8\" (UniqueName: \"kubernetes.io/projected/ffac071d-02fa-4e7a-9f0b-eed903c73d85-kube-api-access-l2dk8\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-m89q4\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.056542 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.056509    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.058360 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.058337    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.076920 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.076892    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.078971 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.078947    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.157582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.157109    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7t24\" (UniqueName: \"kubernetes.io/projected/53501126-73af-4b23-888c-e9d1660de15b-kube-api-access-r7t24\") pod \"limitador-operator-controller-manager-74495c69f8-sppbt\" (UID: \"53501126-73af-4b23-888c-e9d1660de15b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"
Sep 10 15:39:39.157582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.157158    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kshqn\" (UniqueName: \"kubernetes.io/projected/fda158bd-2217-4935-a3e0-32e1f88b009b-kube-api-access-kshqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-btpqq\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:39.157582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.157190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffac071d-02fa-4e7a-9f0b-eed903c73d85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-m89q4\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.157582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.157205    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2dk8\" (UniqueName: \"kubernetes.io/projected/ffac071d-02fa-4e7a-9f0b-eed903c73d85-kube-api-access-l2dk8\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-m89q4\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.157582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.157241    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fda158bd-2217-4935-a3e0-32e1f88b009b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-btpqq\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:39.157582 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.157543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fda158bd-2217-4935-a3e0-32e1f88b009b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-76f8f686bf-btpqq\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:39.157891 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.157688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffac071d-02fa-4e7a-9f0b-eed903c73d85-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-m89q4\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.166728 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.166668    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kshqn\" (UniqueName: \"kubernetes.io/projected/fda158bd-2217-4935-a3e0-32e1f88b009b-kube-api-access-kshqn\") pod \"kuadrant-operator-controller-manager-76f8f686bf-btpqq\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:39.167745 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.167719    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7t24\" (UniqueName: \"kubernetes.io/projected/53501126-73af-4b23-888c-e9d1660de15b-kube-api-access-r7t24\") pod \"limitador-operator-controller-manager-74495c69f8-sppbt\" (UID: \"53501126-73af-4b23-888c-e9d1660de15b\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"
Sep 10 15:39:39.168384 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.168272    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2dk8\" (UniqueName: \"kubernetes.io/projected/ffac071d-02fa-4e7a-9f0b-eed903c73d85-kube-api-access-l2dk8\") pod \"kuadrant-operator-controller-manager-67c5dcb48c-m89q4\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") " pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.219635 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.219610    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.222221 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.222196    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.223875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.223858    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.224907 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.224885    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.227243 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.227220    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.229248 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.229227    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.257672 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.257652    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bgpz\" (UniqueName: \"kubernetes.io/projected/169a3d34-d81a-4746-a1eb-317dd0a5e8eb-kube-api-access-6bgpz\") pod \"169a3d34-d81a-4746-a1eb-317dd0a5e8eb\" (UID: \"169a3d34-d81a-4746-a1eb-317dd0a5e8eb\") "
Sep 10 15:39:39.257783 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.257691    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-extensions-socket-volume\") pod \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") "
Sep 10 15:39:39.257783 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.257718    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fchjd\" (UniqueName: \"kubernetes.io/projected/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-kube-api-access-fchjd\") pod \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\" (UID: \"5c9b2d42-60b3-48df-8f36-ecc395ba6f48\") "
Sep 10 15:39:39.258077 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.258055    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "5c9b2d42-60b3-48df-8f36-ecc395ba6f48" (UID: "5c9b2d42-60b3-48df-8f36-ecc395ba6f48"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:39:39.259936 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.259915    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-kube-api-access-fchjd" (OuterVolumeSpecName: "kube-api-access-fchjd") pod "5c9b2d42-60b3-48df-8f36-ecc395ba6f48" (UID: "5c9b2d42-60b3-48df-8f36-ecc395ba6f48"). InnerVolumeSpecName "kube-api-access-fchjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:39:39.259999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.259940    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/169a3d34-d81a-4746-a1eb-317dd0a5e8eb-kube-api-access-6bgpz" (OuterVolumeSpecName: "kube-api-access-6bgpz") pod "169a3d34-d81a-4746-a1eb-317dd0a5e8eb" (UID: "169a3d34-d81a-4746-a1eb-317dd0a5e8eb"). InnerVolumeSpecName "kube-api-access-6bgpz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:39:39.358968 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.358919    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fchjd\" (UniqueName: \"kubernetes.io/projected/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-kube-api-access-fchjd\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:39:39.358968 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.358964    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bgpz\" (UniqueName: \"kubernetes.io/projected/169a3d34-d81a-4746-a1eb-317dd0a5e8eb-kube-api-access-6bgpz\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:39:39.358968 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.358978    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/5c9b2d42-60b3-48df-8f36-ecc395ba6f48-extensions-socket-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:39:39.373911 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.373883    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.381486 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.381387    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:39.386462 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.386437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"
Sep 10 15:39:39.658378 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.658350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"]
Sep 10 15:39:39.660284 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:39.660260    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffac071d_02fa_4e7a_9f0b_eed903c73d85.slice/crio-c4dfa5c60fdeb6ee3050ee1fa56c4e97a705ec5afded0f4d6cffb370cd28ffaa WatchSource:0}: Error finding container c4dfa5c60fdeb6ee3050ee1fa56c4e97a705ec5afded0f4d6cffb370cd28ffaa: Status 404 returned error can't find the container with id c4dfa5c60fdeb6ee3050ee1fa56c4e97a705ec5afded0f4d6cffb370cd28ffaa
Sep 10 15:39:39.671809 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.671788    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"]
Sep 10 15:39:39.674413 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:39.674390    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda158bd_2217_4935_a3e0_32e1f88b009b.slice/crio-4753f622ddc31a119eb41a1ff1798c04db4264bc9e30ae2db865f0e39cbf5774 WatchSource:0}: Error finding container 4753f622ddc31a119eb41a1ff1798c04db4264bc9e30ae2db865f0e39cbf5774: Status 404 returned error can't find the container with id 4753f622ddc31a119eb41a1ff1798c04db4264bc9e30ae2db865f0e39cbf5774
Sep 10 15:39:39.681744 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.681699    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"]
Sep 10 15:39:39.693827 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:39.693805    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53501126_73af_4b23_888c_e9d1660de15b.slice/crio-3a9e0f9fd36adb25f657c612051b2a01d1c00880c8036c233909b509dd174f1e WatchSource:0}: Error finding container 3a9e0f9fd36adb25f657c612051b2a01d1c00880c8036c233909b509dd174f1e: Status 404 returned error can't find the container with id 3a9e0f9fd36adb25f657c612051b2a01d1c00880c8036c233909b509dd174f1e
Sep 10 15:39:39.787856 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.787821    2577 generic.go:356] "Generic (PLEG): container finished" podID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" containerID="28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6" exitCode=0
Sep 10 15:39:39.788091 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.787885    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.788091 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.787930    2577 scope.go:117] "RemoveContainer" containerID="28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6"
Sep 10 15:39:39.790769 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.790723    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt" event={"ID":"53501126-73af-4b23-888c-e9d1660de15b","Type":"ContainerStarted","Data":"58380697ec5f826596b0a994400d4a1199a80a6a9b74b207401e22b0046cd206"}
Sep 10 15:39:39.790879 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.790809    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt" event={"ID":"53501126-73af-4b23-888c-e9d1660de15b","Type":"ContainerStarted","Data":"3a9e0f9fd36adb25f657c612051b2a01d1c00880c8036c233909b509dd174f1e"}
Sep 10 15:39:39.791266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.791122    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"
Sep 10 15:39:39.791266 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.791142    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.793332 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.793306    2577 generic.go:356] "Generic (PLEG): container finished" podID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" containerID="780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765" exitCode=0
Sep 10 15:39:39.793427 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.793373    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.793488 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.793473    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.796429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.796335    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4" event={"ID":"ffac071d-02fa-4e7a-9f0b-eed903c73d85","Type":"ContainerStarted","Data":"cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae"}
Sep 10 15:39:39.796429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.796379    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4" event={"ID":"ffac071d-02fa-4e7a-9f0b-eed903c73d85","Type":"ContainerStarted","Data":"c4dfa5c60fdeb6ee3050ee1fa56c4e97a705ec5afded0f4d6cffb370cd28ffaa"}
Sep 10 15:39:39.796429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.796410    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:39.799310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.799178    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq" event={"ID":"fda158bd-2217-4935-a3e0-32e1f88b009b","Type":"ContainerStarted","Data":"b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa"}
Sep 10 15:39:39.799310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.799230    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq" event={"ID":"fda158bd-2217-4935-a3e0-32e1f88b009b","Type":"ContainerStarted","Data":"4753f622ddc31a119eb41a1ff1798c04db4264bc9e30ae2db865f0e39cbf5774"}
Sep 10 15:39:39.799471 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.799401    2577 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr" container="manager"
Sep 10 15:39:39.802721 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.802702    2577 scope.go:117] "RemoveContainer" containerID="28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6"
Sep 10 15:39:39.803410 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.803276    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6\": container with ID starting with 28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6 not found: ID does not exist" containerID="28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6"
Sep 10 15:39:39.803410 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.803311    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6"} err="failed to get container status \"28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6\": rpc error: code = NotFound desc = could not find container \"28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6\": container with ID starting with 28cee36028f22f11e29b6c08d73cea1acbd4ddc8552193860076c1cdfc4a49d6 not found: ID does not exist"
Sep 10 15:39:39.803410 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.803334    2577 scope.go:117] "RemoveContainer" containerID="780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765"
Sep 10 15:39:39.812499 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.812483    2577 scope.go:117] "RemoveContainer" containerID="780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765"
Sep 10 15:39:39.812773 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.812738    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765\": container with ID starting with 780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765 not found: ID does not exist" containerID="780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765"
Sep 10 15:39:39.812826 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.812781    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765"} err="failed to get container status \"780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765\": rpc error: code = NotFound desc = could not find container \"780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765\": container with ID starting with 780d0a528cb7f9a63ff16090813d258775bee152bbab18ff71235394f732f765 not found: ID does not exist"
Sep 10 15:39:39.828564 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.828524    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt" podStartSLOduration=1.828507876 podStartE2EDuration="1.828507876s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:39.826920574 +0000 UTC m=+423.441047479" watchObservedRunningTime="2026-09-10 15:39:39.828507876 +0000 UTC m=+423.442634844"
Sep 10 15:39:39.829246 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.829221    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.831392 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.831368    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:39.833710 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.833689    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:39.863797 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.863733    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq" podStartSLOduration=1.8637195260000001 podStartE2EDuration="1.863719526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:39.861715151 +0000 UTC m=+423.475842055" watchObservedRunningTime="2026-09-10 15:39:39.863719526 +0000 UTC m=+423.477846430"
Sep 10 15:39:39.887764 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.887726    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:39.887975 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:39.887933    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4" podStartSLOduration=1.887916216 podStartE2EDuration="1.887916216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:39.88586949 +0000 UTC m=+423.499996384" watchObservedRunningTime="2026-09-10 15:39:39.887916216 +0000 UTC m=+423.502043121"
Sep 10 15:39:39.889923 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:39.889894    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:40.809987 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:40.809948    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" containerName="manager" containerID="cri-o://672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898" gracePeriod=2
Sep 10 15:39:40.810429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:40.810090    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:40.812249 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:40.812227    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-76f8f686bf-kc2qr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-kc2qr"
Sep 10 15:39:40.814315 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:40.814191    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-qxmrn\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-qxmrn"
Sep 10 15:39:40.816285 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:40.816258    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:40.907841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:40.907809    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="169a3d34-d81a-4746-a1eb-317dd0a5e8eb" path="/var/lib/kubelet/pods/169a3d34-d81a-4746-a1eb-317dd0a5e8eb/volumes"
Sep 10 15:39:40.908685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:40.908644    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c9b2d42-60b3-48df-8f36-ecc395ba6f48" path="/var/lib/kubelet/pods/5c9b2d42-60b3-48df-8f36-ecc395ba6f48/volumes"
Sep 10 15:39:41.045521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.045496    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:41.073797 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.073717    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-extensions-socket-volume\") pod \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") "
Sep 10 15:39:41.073797 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.073783    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f84tp\" (UniqueName: \"kubernetes.io/projected/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-kube-api-access-f84tp\") pod \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\" (UID: \"74f77ff1-2ffa-4d94-ae2b-c5751acf6422\") "
Sep 10 15:39:41.074143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.074124    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "74f77ff1-2ffa-4d94-ae2b-c5751acf6422" (UID: "74f77ff1-2ffa-4d94-ae2b-c5751acf6422"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:39:41.075970 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.075944    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-kube-api-access-f84tp" (OuterVolumeSpecName: "kube-api-access-f84tp") pod "74f77ff1-2ffa-4d94-ae2b-c5751acf6422" (UID: "74f77ff1-2ffa-4d94-ae2b-c5751acf6422"). InnerVolumeSpecName "kube-api-access-f84tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:39:41.174533 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.174497    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-extensions-socket-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:39:41.174533 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.174524    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f84tp\" (UniqueName: \"kubernetes.io/projected/74f77ff1-2ffa-4d94-ae2b-c5751acf6422-kube-api-access-f84tp\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:39:41.817289 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.817251    2577 generic.go:356] "Generic (PLEG): container finished" podID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" containerID="672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898" exitCode=0
Sep 10 15:39:41.817709 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.817296    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:41.817709 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.817341    2577 scope.go:117] "RemoveContainer" containerID="672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898"
Sep 10 15:39:41.827245 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.827226    2577 scope.go:117] "RemoveContainer" containerID="672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898"
Sep 10 15:39:41.827585 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:41.827560    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898\": container with ID starting with 672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898 not found: ID does not exist" containerID="672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898"
Sep 10 15:39:41.827636 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:41.827598    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898"} err="failed to get container status \"672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898\": rpc error: code = NotFound desc = could not find container \"672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898\": container with ID starting with 672864ba10746fd0353a570651ea02efd11176d4f8955e3d0ee5efa4159b1898 not found: ID does not exist"
Sep 10 15:39:41.828545 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:41.828519    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-67c5dcb48c-lzfqr\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-lzfqr"
Sep 10 15:39:42.907367 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:42.907325    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="74f77ff1-2ffa-4d94-ae2b-c5751acf6422" path="/var/lib/kubelet/pods/74f77ff1-2ffa-4d94-ae2b-c5751acf6422/volumes"
Sep 10 15:39:50.812514 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:50.812485    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:39:50.812912 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:50.812551    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-sppbt"
Sep 10 15:39:50.894573 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:50.894534    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"]
Sep 10 15:39:50.894827 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:50.894771    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq" podUID="fda158bd-2217-4935-a3e0-32e1f88b009b" containerName="manager" containerID="cri-o://b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa" gracePeriod=10
Sep 10 15:39:50.896905 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:50.896882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:51.137449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.137427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:51.252166 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.252127    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fda158bd-2217-4935-a3e0-32e1f88b009b-extensions-socket-volume\") pod \"fda158bd-2217-4935-a3e0-32e1f88b009b\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") "
Sep 10 15:39:51.252344 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.252185    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kshqn\" (UniqueName: \"kubernetes.io/projected/fda158bd-2217-4935-a3e0-32e1f88b009b-kube-api-access-kshqn\") pod \"fda158bd-2217-4935-a3e0-32e1f88b009b\" (UID: \"fda158bd-2217-4935-a3e0-32e1f88b009b\") "
Sep 10 15:39:51.252605 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.252580    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fda158bd-2217-4935-a3e0-32e1f88b009b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "fda158bd-2217-4935-a3e0-32e1f88b009b" (UID: "fda158bd-2217-4935-a3e0-32e1f88b009b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:39:51.254421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.254399    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda158bd-2217-4935-a3e0-32e1f88b009b-kube-api-access-kshqn" (OuterVolumeSpecName: "kube-api-access-kshqn") pod "fda158bd-2217-4935-a3e0-32e1f88b009b" (UID: "fda158bd-2217-4935-a3e0-32e1f88b009b"). InnerVolumeSpecName "kube-api-access-kshqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:39:51.307889 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.307856    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"]
Sep 10 15:39:51.314799 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.308582    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda158bd-2217-4935-a3e0-32e1f88b009b" containerName="manager"
Sep 10 15:39:51.314799 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.308642    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda158bd-2217-4935-a3e0-32e1f88b009b" containerName="manager"
Sep 10 15:39:51.314799 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.308940    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda158bd-2217-4935-a3e0-32e1f88b009b" containerName="manager"
Sep 10 15:39:51.316391 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.316367    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.332891 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.332870    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"]
Sep 10 15:39:51.352626 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.352591    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p4pb\" (UniqueName: \"kubernetes.io/projected/d904b35c-d482-4e27-b14f-6031df978c0a-kube-api-access-5p4pb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ncllz\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.352793 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.352651    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d904b35c-d482-4e27-b14f-6031df978c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ncllz\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.352793 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.352704    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/fda158bd-2217-4935-a3e0-32e1f88b009b-extensions-socket-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:39:51.352793 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.352714    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kshqn\" (UniqueName: \"kubernetes.io/projected/fda158bd-2217-4935-a3e0-32e1f88b009b-kube-api-access-kshqn\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:39:51.453269 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.453175    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d904b35c-d482-4e27-b14f-6031df978c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ncllz\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.453269 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.453252    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p4pb\" (UniqueName: \"kubernetes.io/projected/d904b35c-d482-4e27-b14f-6031df978c0a-kube-api-access-5p4pb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ncllz\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.453556 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.453527    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d904b35c-d482-4e27-b14f-6031df978c0a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ncllz\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.461698 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.461673    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p4pb\" (UniqueName: \"kubernetes.io/projected/d904b35c-d482-4e27-b14f-6031df978c0a-kube-api-access-5p4pb\") pod \"kuadrant-operator-controller-manager-5769f99d9f-ncllz\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.626012 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.625975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:51.873378 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.873342    2577 generic.go:356] "Generic (PLEG): container finished" podID="fda158bd-2217-4935-a3e0-32e1f88b009b" containerID="b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa" exitCode=0
Sep 10 15:39:51.873862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.873427    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"
Sep 10 15:39:51.873862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.873425    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq" event={"ID":"fda158bd-2217-4935-a3e0-32e1f88b009b","Type":"ContainerDied","Data":"b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa"}
Sep 10 15:39:51.873862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.873570    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq" event={"ID":"fda158bd-2217-4935-a3e0-32e1f88b009b","Type":"ContainerDied","Data":"4753f622ddc31a119eb41a1ff1798c04db4264bc9e30ae2db865f0e39cbf5774"}
Sep 10 15:39:51.873862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.873593    2577 scope.go:117] "RemoveContainer" containerID="b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa"
Sep 10 15:39:51.883094 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.883077    2577 scope.go:117] "RemoveContainer" containerID="b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa"
Sep 10 15:39:51.883366 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:39:51.883348    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa\": container with ID starting with b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa not found: ID does not exist" containerID="b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa"
Sep 10 15:39:51.883410 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.883376    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa"} err="failed to get container status \"b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa\": rpc error: code = NotFound desc = could not find container \"b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa\": container with ID starting with b491502cc32c6a17e11334693c2136cebdd45a6cc8158866a5a914bd936b42fa not found: ID does not exist"
Sep 10 15:39:51.900369 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.900344    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"]
Sep 10 15:39:51.907372 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.907350    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"]
Sep 10 15:39:51.910080 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:39:51.910051    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd904b35c_d482_4e27_b14f_6031df978c0a.slice/crio-c7eb52b7b4b63321a4a8ca277d13c96c6c65bed53c6d6af297601f4c7663f9d0 WatchSource:0}: Error finding container c7eb52b7b4b63321a4a8ca277d13c96c6c65bed53c6d6af297601f4c7663f9d0: Status 404 returned error can't find the container with id c7eb52b7b4b63321a4a8ca277d13c96c6c65bed53c6d6af297601f4c7663f9d0
Sep 10 15:39:51.911875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:51.911857    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-76f8f686bf-btpqq"]
Sep 10 15:39:52.880079 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:52.880043    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz" event={"ID":"d904b35c-d482-4e27-b14f-6031df978c0a","Type":"ContainerStarted","Data":"9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903"}
Sep 10 15:39:52.880506 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:52.880094    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz" event={"ID":"d904b35c-d482-4e27-b14f-6031df978c0a","Type":"ContainerStarted","Data":"c7eb52b7b4b63321a4a8ca277d13c96c6c65bed53c6d6af297601f4c7663f9d0"}
Sep 10 15:39:52.880506 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:52.880141    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:39:52.902851 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:52.902791    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz" podStartSLOduration=1.902770271 podStartE2EDuration="1.902770271s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:39:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:39:52.901171283 +0000 UTC m=+436.515298186" watchObservedRunningTime="2026-09-10 15:39:52.902770271 +0000 UTC m=+436.516897170"
Sep 10 15:39:52.908273 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:39:52.908248    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda158bd-2217-4935-a3e0-32e1f88b009b" path="/var/lib/kubelet/pods/fda158bd-2217-4935-a3e0-32e1f88b009b/volumes"
Sep 10 15:40:03.889273 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:03.889187    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 15:40:03.943547 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:03.943515    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"]
Sep 10 15:40:03.943801 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:03.943775    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4" podUID="ffac071d-02fa-4e7a-9f0b-eed903c73d85" containerName="manager" containerID="cri-o://cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae" gracePeriod=10
Sep 10 15:40:04.186046 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.186020    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:40:04.258939 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.258903    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2dk8\" (UniqueName: \"kubernetes.io/projected/ffac071d-02fa-4e7a-9f0b-eed903c73d85-kube-api-access-l2dk8\") pod \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") "
Sep 10 15:40:04.259112 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.258952    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffac071d-02fa-4e7a-9f0b-eed903c73d85-extensions-socket-volume\") pod \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\" (UID: \"ffac071d-02fa-4e7a-9f0b-eed903c73d85\") "
Sep 10 15:40:04.259371 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.259345    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffac071d-02fa-4e7a-9f0b-eed903c73d85-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "ffac071d-02fa-4e7a-9f0b-eed903c73d85" (UID: "ffac071d-02fa-4e7a-9f0b-eed903c73d85"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:40:04.261191 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.261167    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffac071d-02fa-4e7a-9f0b-eed903c73d85-kube-api-access-l2dk8" (OuterVolumeSpecName: "kube-api-access-l2dk8") pod "ffac071d-02fa-4e7a-9f0b-eed903c73d85" (UID: "ffac071d-02fa-4e7a-9f0b-eed903c73d85"). InnerVolumeSpecName "kube-api-access-l2dk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:04.359910 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.359866    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/ffac071d-02fa-4e7a-9f0b-eed903c73d85-extensions-socket-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.359910 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.359902    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2dk8\" (UniqueName: \"kubernetes.io/projected/ffac071d-02fa-4e7a-9f0b-eed903c73d85-kube-api-access-l2dk8\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:04.943984 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.943946    2577 generic.go:356] "Generic (PLEG): container finished" podID="ffac071d-02fa-4e7a-9f0b-eed903c73d85" containerID="cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae" exitCode=0
Sep 10 15:40:04.944418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.943993    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"
Sep 10 15:40:04.944418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.944036    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4" event={"ID":"ffac071d-02fa-4e7a-9f0b-eed903c73d85","Type":"ContainerDied","Data":"cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae"}
Sep 10 15:40:04.944418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.944102    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4" event={"ID":"ffac071d-02fa-4e7a-9f0b-eed903c73d85","Type":"ContainerDied","Data":"c4dfa5c60fdeb6ee3050ee1fa56c4e97a705ec5afded0f4d6cffb370cd28ffaa"}
Sep 10 15:40:04.944418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.944136    2577 scope.go:117] "RemoveContainer" containerID="cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae"
Sep 10 15:40:04.952920 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.952906    2577 scope.go:117] "RemoveContainer" containerID="cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae"
Sep 10 15:40:04.953243 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:40:04.953225    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae\": container with ID starting with cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae not found: ID does not exist" containerID="cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae"
Sep 10 15:40:04.953311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.953256    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae"} err="failed to get container status \"cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae\": rpc error: code = NotFound desc = could not find container \"cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae\": container with ID starting with cc60be013c85d5f21a329f4303dca37c74126c132f14e238a5497dad9e2d65ae not found: ID does not exist"
Sep 10 15:40:04.966224 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.966192    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"]
Sep 10 15:40:04.976657 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:04.976626    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-67c5dcb48c-m89q4"]
Sep 10 15:40:06.907468 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:06.907427    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ffac071d-02fa-4e7a-9f0b-eed903c73d85" path="/var/lib/kubelet/pods/ffac071d-02fa-4e7a-9f0b-eed903c73d85/volumes"
Sep 10 15:40:20.554445 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.554413    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kp2fg"]
Sep 10 15:40:20.554945 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.554847    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ffac071d-02fa-4e7a-9f0b-eed903c73d85" containerName="manager"
Sep 10 15:40:20.554945 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.554864    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="ffac071d-02fa-4e7a-9f0b-eed903c73d85" containerName="manager"
Sep 10 15:40:20.555056 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.554959    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="ffac071d-02fa-4e7a-9f0b-eed903c73d85" containerName="manager"
Sep 10 15:40:20.559082 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.559061    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:20.561671 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.561652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 15:40:20.561807 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.561655    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n4c6m\""
Sep 10 15:40:20.566264 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.566243    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kp2fg"]
Sep 10 15:40:20.650027 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.649990    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kp2fg"]
Sep 10 15:40:20.696846 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.696813    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20386533-47e7-4bbf-b836-c3c2c536264a-config-file\") pod \"limitador-limitador-8588749977-kp2fg\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") " pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:20.697004 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.696888    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xt77x\" (UniqueName: \"kubernetes.io/projected/20386533-47e7-4bbf-b836-c3c2c536264a-kube-api-access-xt77x\") pod \"limitador-limitador-8588749977-kp2fg\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") " pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:20.797324 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.797289    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xt77x\" (UniqueName: \"kubernetes.io/projected/20386533-47e7-4bbf-b836-c3c2c536264a-kube-api-access-xt77x\") pod \"limitador-limitador-8588749977-kp2fg\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") " pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:20.797492 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.797340    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20386533-47e7-4bbf-b836-c3c2c536264a-config-file\") pod \"limitador-limitador-8588749977-kp2fg\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") " pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:20.798022 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.798000    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20386533-47e7-4bbf-b836-c3c2c536264a-config-file\") pod \"limitador-limitador-8588749977-kp2fg\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") " pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:20.808784 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.808698    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xt77x\" (UniqueName: \"kubernetes.io/projected/20386533-47e7-4bbf-b836-c3c2c536264a-kube-api-access-xt77x\") pod \"limitador-limitador-8588749977-kp2fg\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") " pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:20.870683 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:20.870657    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:21.147409 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.147387    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kp2fg"]
Sep 10 15:40:21.149415 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:40:21.149385    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20386533_47e7_4bbf_b836_c3c2c536264a.slice/crio-824a70904c2c507893c74b359d010fff55133d40863eb0111e46a3145d970298 WatchSource:0}: Error finding container 824a70904c2c507893c74b359d010fff55133d40863eb0111e46a3145d970298: Status 404 returned error can't find the container with id 824a70904c2c507893c74b359d010fff55133d40863eb0111e46a3145d970298
Sep 10 15:40:21.367724 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.367686    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v86xx"]
Sep 10 15:40:21.371996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.371980    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx"
Sep 10 15:40:21.374402 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.374380    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fhb8b\""
Sep 10 15:40:21.379079 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.379053    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v86xx"]
Sep 10 15:40:21.503103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.503019    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x442\" (UniqueName: \"kubernetes.io/projected/3234542c-3391-4a38-b5d8-19fc50ec52fb-kube-api-access-6x442\") pod \"authorino-7bcfd8f9cd-v86xx\" (UID: \"3234542c-3391-4a38-b5d8-19fc50ec52fb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx"
Sep 10 15:40:21.560236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.560205    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-mcmrb"]
Sep 10 15:40:21.563469 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.563449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mcmrb"
Sep 10 15:40:21.568600 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.568574    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mcmrb"]
Sep 10 15:40:21.603717 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.603683    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x442\" (UniqueName: \"kubernetes.io/projected/3234542c-3391-4a38-b5d8-19fc50ec52fb-kube-api-access-6x442\") pod \"authorino-7bcfd8f9cd-v86xx\" (UID: \"3234542c-3391-4a38-b5d8-19fc50ec52fb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx"
Sep 10 15:40:21.615742 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.615715    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x442\" (UniqueName: \"kubernetes.io/projected/3234542c-3391-4a38-b5d8-19fc50ec52fb-kube-api-access-6x442\") pod \"authorino-7bcfd8f9cd-v86xx\" (UID: \"3234542c-3391-4a38-b5d8-19fc50ec52fb\") " pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx"
Sep 10 15:40:21.682660 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.682624    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx"
Sep 10 15:40:21.705097 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.705062    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghfqg\" (UniqueName: \"kubernetes.io/projected/0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f-kube-api-access-ghfqg\") pod \"authorino-59c974bd48-mcmrb\" (UID: \"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f\") " pod="kuadrant-system/authorino-59c974bd48-mcmrb"
Sep 10 15:40:21.806526 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.806419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghfqg\" (UniqueName: \"kubernetes.io/projected/0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f-kube-api-access-ghfqg\") pod \"authorino-59c974bd48-mcmrb\" (UID: \"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f\") " pod="kuadrant-system/authorino-59c974bd48-mcmrb"
Sep 10 15:40:21.815240 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.815208    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghfqg\" (UniqueName: \"kubernetes.io/projected/0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f-kube-api-access-ghfqg\") pod \"authorino-59c974bd48-mcmrb\" (UID: \"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f\") " pod="kuadrant-system/authorino-59c974bd48-mcmrb"
Sep 10 15:40:21.873811 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.873770    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mcmrb"
Sep 10 15:40:21.956318 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:21.956287    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v86xx"]
Sep 10 15:40:21.959721 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:40:21.959693    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3234542c_3391_4a38_b5d8_19fc50ec52fb.slice/crio-8f0067228225cc65ac5d1c18c64e2d286b124ebe9c9779a325eea565cc759886 WatchSource:0}: Error finding container 8f0067228225cc65ac5d1c18c64e2d286b124ebe9c9779a325eea565cc759886: Status 404 returned error can't find the container with id 8f0067228225cc65ac5d1c18c64e2d286b124ebe9c9779a325eea565cc759886
Sep 10 15:40:22.031640 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:22.031598    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx" event={"ID":"3234542c-3391-4a38-b5d8-19fc50ec52fb","Type":"ContainerStarted","Data":"8f0067228225cc65ac5d1c18c64e2d286b124ebe9c9779a325eea565cc759886"}
Sep 10 15:40:22.033140 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:22.033108    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg" event={"ID":"20386533-47e7-4bbf-b836-c3c2c536264a","Type":"ContainerStarted","Data":"824a70904c2c507893c74b359d010fff55133d40863eb0111e46a3145d970298"}
Sep 10 15:40:22.151405 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:22.151378    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mcmrb"]
Sep 10 15:40:22.239241 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:40:22.239201    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0a04acc2_f22f_4cc5_9d95_ac0df80c7c6f.slice/crio-fd610ae9b6b917327e0cf7106ba60a391c36d891c6002b928d0ed4930daa4b9f WatchSource:0}: Error finding container fd610ae9b6b917327e0cf7106ba60a391c36d891c6002b928d0ed4930daa4b9f: Status 404 returned error can't find the container with id fd610ae9b6b917327e0cf7106ba60a391c36d891c6002b928d0ed4930daa4b9f
Sep 10 15:40:23.041636 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:23.041596    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg" event={"ID":"20386533-47e7-4bbf-b836-c3c2c536264a","Type":"ContainerStarted","Data":"c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846"}
Sep 10 15:40:23.042130 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:23.041698    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:23.043044 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:23.043020    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mcmrb" event={"ID":"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f","Type":"ContainerStarted","Data":"fd610ae9b6b917327e0cf7106ba60a391c36d891c6002b928d0ed4930daa4b9f"}
Sep 10 15:40:23.059683 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:23.059632    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg" podStartSLOduration=1.91440775 podStartE2EDuration="3.059611219s" totalImagesPullingTime="1.145203469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:40:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:40:23.057711724 +0000 UTC m=+466.671838622" watchObservedRunningTime="2026-09-10 15:40:23.059611219 +0000 UTC m=+466.673738193"
Sep 10 15:40:24.049979 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:24.049882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx" event={"ID":"3234542c-3391-4a38-b5d8-19fc50ec52fb","Type":"ContainerStarted","Data":"7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb"}
Sep 10 15:40:24.051415 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:24.051391    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mcmrb" event={"ID":"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f","Type":"ContainerStarted","Data":"2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693"}
Sep 10 15:40:24.079386 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:24.079314    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-mcmrb" podStartSLOduration=1.5627237680000001 podStartE2EDuration="3.079294509s" totalImagesPullingTime="1.516570741s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:40:24.078211559 +0000 UTC m=+467.692338463" watchObservedRunningTime="2026-09-10 15:40:24.079294509 +0000 UTC m=+467.693421414"
Sep 10 15:40:24.081588 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:24.081543    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx" podStartSLOduration=1.290051813 podStartE2EDuration="3.081527877s" totalImagesPullingTime="1.791476064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:40:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:40:24.064067843 +0000 UTC m=+467.678194748" watchObservedRunningTime="2026-09-10 15:40:24.081527877 +0000 UTC m=+467.695654782"
Sep 10 15:40:24.109225 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:24.109191    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v86xx"]
Sep 10 15:40:26.062301 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:26.062265    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx" podUID="3234542c-3391-4a38-b5d8-19fc50ec52fb" containerName="authorino" containerID="cri-o://7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb" gracePeriod=30
Sep 10 15:40:26.297355 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:26.297330    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx"
Sep 10 15:40:26.445496 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:26.445466    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6x442\" (UniqueName: \"kubernetes.io/projected/3234542c-3391-4a38-b5d8-19fc50ec52fb-kube-api-access-6x442\") pod \"3234542c-3391-4a38-b5d8-19fc50ec52fb\" (UID: \"3234542c-3391-4a38-b5d8-19fc50ec52fb\") "
Sep 10 15:40:26.447739 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:26.447700    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3234542c-3391-4a38-b5d8-19fc50ec52fb-kube-api-access-6x442" (OuterVolumeSpecName: "kube-api-access-6x442") pod "3234542c-3391-4a38-b5d8-19fc50ec52fb" (UID: "3234542c-3391-4a38-b5d8-19fc50ec52fb"). InnerVolumeSpecName "kube-api-access-6x442". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:26.546620 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:26.546588    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6x442\" (UniqueName: \"kubernetes.io/projected/3234542c-3391-4a38-b5d8-19fc50ec52fb-kube-api-access-6x442\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:27.068662 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.068632    2577 generic.go:356] "Generic (PLEG): container finished" podID="3234542c-3391-4a38-b5d8-19fc50ec52fb" containerID="7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb" exitCode=0
Sep 10 15:40:27.069104 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.068679    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx"
Sep 10 15:40:27.069104 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.068721    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx" event={"ID":"3234542c-3391-4a38-b5d8-19fc50ec52fb","Type":"ContainerDied","Data":"7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb"}
Sep 10 15:40:27.069104 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.068778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-v86xx" event={"ID":"3234542c-3391-4a38-b5d8-19fc50ec52fb","Type":"ContainerDied","Data":"8f0067228225cc65ac5d1c18c64e2d286b124ebe9c9779a325eea565cc759886"}
Sep 10 15:40:27.069104 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.068796    2577 scope.go:117] "RemoveContainer" containerID="7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb"
Sep 10 15:40:27.078017 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.077998    2577 scope.go:117] "RemoveContainer" containerID="7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb"
Sep 10 15:40:27.078351 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:40:27.078330    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb\": container with ID starting with 7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb not found: ID does not exist" containerID="7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb"
Sep 10 15:40:27.078397 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.078361    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb"} err="failed to get container status \"7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb\": rpc error: code = NotFound desc = could not find container \"7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb\": container with ID starting with 7eca145fa9cfcd34f1b939f000331c267d89e322f54dd9009a602f262e330efb not found: ID does not exist"
Sep 10 15:40:27.083355 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.083329    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v86xx"]
Sep 10 15:40:27.086148 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:27.086126    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-v86xx"]
Sep 10 15:40:28.907117 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:28.907085    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3234542c-3391-4a38-b5d8-19fc50ec52fb" path="/var/lib/kubelet/pods/3234542c-3391-4a38-b5d8-19fc50ec52fb/volumes"
Sep 10 15:40:34.052374 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:34.052345    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:35.128205 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:35.128177    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kp2fg"]
Sep 10 15:40:35.128683 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:35.128385    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg" podUID="20386533-47e7-4bbf-b836-c3c2c536264a" containerName="limitador" containerID="cri-o://c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846" gracePeriod=30
Sep 10 15:40:36.082736 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.082711    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:36.117230 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.117199    2577 generic.go:356] "Generic (PLEG): container finished" podID="20386533-47e7-4bbf-b836-c3c2c536264a" containerID="c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846" exitCode=0
Sep 10 15:40:36.117421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.117242    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg"
Sep 10 15:40:36.117421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.117286    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg" event={"ID":"20386533-47e7-4bbf-b836-c3c2c536264a","Type":"ContainerDied","Data":"c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846"}
Sep 10 15:40:36.117421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.117344    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-8588749977-kp2fg" event={"ID":"20386533-47e7-4bbf-b836-c3c2c536264a","Type":"ContainerDied","Data":"824a70904c2c507893c74b359d010fff55133d40863eb0111e46a3145d970298"}
Sep 10 15:40:36.117421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.117369    2577 scope.go:117] "RemoveContainer" containerID="c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846"
Sep 10 15:40:36.125955 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.125939    2577 scope.go:117] "RemoveContainer" containerID="c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846"
Sep 10 15:40:36.126227 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:40:36.126208    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846\": container with ID starting with c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846 not found: ID does not exist" containerID="c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846"
Sep 10 15:40:36.126285 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.126238    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846"} err="failed to get container status \"c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846\": rpc error: code = NotFound desc = could not find container \"c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846\": container with ID starting with c6868f13d02493d1597eccd6b34c0da7ed8a8d95c72e7ce52f74cdf991fa5846 not found: ID does not exist"
Sep 10 15:40:36.221822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.221708    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20386533-47e7-4bbf-b836-c3c2c536264a-config-file\") pod \"20386533-47e7-4bbf-b836-c3c2c536264a\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") "
Sep 10 15:40:36.221822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.221778    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xt77x\" (UniqueName: \"kubernetes.io/projected/20386533-47e7-4bbf-b836-c3c2c536264a-kube-api-access-xt77x\") pod \"20386533-47e7-4bbf-b836-c3c2c536264a\" (UID: \"20386533-47e7-4bbf-b836-c3c2c536264a\") "
Sep 10 15:40:36.222255 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.222044    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20386533-47e7-4bbf-b836-c3c2c536264a-config-file" (OuterVolumeSpecName: "config-file") pod "20386533-47e7-4bbf-b836-c3c2c536264a" (UID: "20386533-47e7-4bbf-b836-c3c2c536264a"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:40:36.224029 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.223992    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20386533-47e7-4bbf-b836-c3c2c536264a-kube-api-access-xt77x" (OuterVolumeSpecName: "kube-api-access-xt77x") pod "20386533-47e7-4bbf-b836-c3c2c536264a" (UID: "20386533-47e7-4bbf-b836-c3c2c536264a"). InnerVolumeSpecName "kube-api-access-xt77x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:36.322417 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.322378    2577 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/20386533-47e7-4bbf-b836-c3c2c536264a-config-file\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:36.322417 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.322409    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xt77x\" (UniqueName: \"kubernetes.io/projected/20386533-47e7-4bbf-b836-c3c2c536264a-kube-api-access-xt77x\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:36.439942 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.439910    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kp2fg"]
Sep 10 15:40:36.443408 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.443385    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-8588749977-kp2fg"]
Sep 10 15:40:36.908727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:36.908699    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20386533-47e7-4bbf-b836-c3c2c536264a" path="/var/lib/kubelet/pods/20386533-47e7-4bbf-b836-c3c2c536264a/volumes"
Sep 10 15:40:41.304530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.304501    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"]
Sep 10 15:40:41.304938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.304816    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20386533-47e7-4bbf-b836-c3c2c536264a" containerName="limitador"
Sep 10 15:40:41.304938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.304828    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20386533-47e7-4bbf-b836-c3c2c536264a" containerName="limitador"
Sep 10 15:40:41.304938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.304840    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3234542c-3391-4a38-b5d8-19fc50ec52fb" containerName="authorino"
Sep 10 15:40:41.304938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.304845    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3234542c-3391-4a38-b5d8-19fc50ec52fb" containerName="authorino"
Sep 10 15:40:41.304938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.304901    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20386533-47e7-4bbf-b836-c3c2c536264a" containerName="limitador"
Sep 10 15:40:41.304938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.304911    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3234542c-3391-4a38-b5d8-19fc50ec52fb" containerName="authorino"
Sep 10 15:40:41.309444 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.309423    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.312300 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.312278    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Sep 10 15:40:41.312423 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.312316    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-7kjtx\""
Sep 10 15:40:41.312667 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.312648    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 15:40:41.312778 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.312654    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 15:40:41.318356 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.318334    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"]
Sep 10 15:40:41.360825 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.360799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qznfx\" (UniqueName: \"kubernetes.io/projected/f56c561c-4da8-4e21-8fd0-3fef3671aa8b-kube-api-access-qznfx\") pod \"postgres-69fb4594cf-kplx7\" (UID: \"f56c561c-4da8-4e21-8fd0-3fef3671aa8b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.360995 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.360835    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f56c561c-4da8-4e21-8fd0-3fef3671aa8b-data\") pod \"postgres-69fb4594cf-kplx7\" (UID: \"f56c561c-4da8-4e21-8fd0-3fef3671aa8b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.462222 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.462190    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qznfx\" (UniqueName: \"kubernetes.io/projected/f56c561c-4da8-4e21-8fd0-3fef3671aa8b-kube-api-access-qznfx\") pod \"postgres-69fb4594cf-kplx7\" (UID: \"f56c561c-4da8-4e21-8fd0-3fef3671aa8b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.462398 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.462233    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f56c561c-4da8-4e21-8fd0-3fef3671aa8b-data\") pod \"postgres-69fb4594cf-kplx7\" (UID: \"f56c561c-4da8-4e21-8fd0-3fef3671aa8b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.462562 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.462547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/f56c561c-4da8-4e21-8fd0-3fef3671aa8b-data\") pod \"postgres-69fb4594cf-kplx7\" (UID: \"f56c561c-4da8-4e21-8fd0-3fef3671aa8b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.471123 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.471100    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qznfx\" (UniqueName: \"kubernetes.io/projected/f56c561c-4da8-4e21-8fd0-3fef3671aa8b-kube-api-access-qznfx\") pod \"postgres-69fb4594cf-kplx7\" (UID: \"f56c561c-4da8-4e21-8fd0-3fef3671aa8b\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.619347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.619302    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:41.891820 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:41.891798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"]
Sep 10 15:40:41.893876 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:40:41.893846    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56c561c_4da8_4e21_8fd0_3fef3671aa8b.slice/crio-05697c94f9425c64e030641813cc05ec11e1329fd1e87a56e3b9d4d19d3beb6f WatchSource:0}: Error finding container 05697c94f9425c64e030641813cc05ec11e1329fd1e87a56e3b9d4d19d3beb6f: Status 404 returned error can't find the container with id 05697c94f9425c64e030641813cc05ec11e1329fd1e87a56e3b9d4d19d3beb6f
Sep 10 15:40:42.148387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:42.148292    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7" event={"ID":"f56c561c-4da8-4e21-8fd0-3fef3671aa8b","Type":"ContainerStarted","Data":"05697c94f9425c64e030641813cc05ec11e1329fd1e87a56e3b9d4d19d3beb6f"}
Sep 10 15:40:47.182637 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:47.182605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7" event={"ID":"f56c561c-4da8-4e21-8fd0-3fef3671aa8b","Type":"ContainerStarted","Data":"4c5b2dd98b69d8ca799018b83be444735f1ec121a72b23bfbf9031c841fb519f"}
Sep 10 15:40:47.183113 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:47.182659    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:47.200176 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:47.200131    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7" podStartSLOduration=1.360960477 podStartE2EDuration="6.20011543s" totalImagesPullingTime="4.839154953s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:40:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:40:47.19877902 +0000 UTC m=+490.812905924" watchObservedRunningTime="2026-09-10 15:40:47.20011543 +0000 UTC m=+490.814242378"
Sep 10 15:40:53.218065 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.218038    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-kplx7"
Sep 10 15:40:53.765321 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.765287    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-ltw8w"]
Sep 10 15:40:53.771371 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.771350    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltw8w"
Sep 10 15:40:53.778247 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.777916    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltw8w"]
Sep 10 15:40:53.834205 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.834171    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltw8w"]
Sep 10 15:40:53.834592 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:40:53.834565    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-wr2kn], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-ltw8w" podUID="b4a58c25-d6a8-4178-9930-2633f4eb65de"
Sep 10 15:40:53.864173 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.864136    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fzs2q"]
Sep 10 15:40:53.866274 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.866249    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr2kn\" (UniqueName: \"kubernetes.io/projected/b4a58c25-d6a8-4178-9930-2633f4eb65de-kube-api-access-wr2kn\") pod \"authorino-557f876f69-ltw8w\" (UID: \"b4a58c25-d6a8-4178-9930-2633f4eb65de\") " pod="kuadrant-system/authorino-557f876f69-ltw8w"
Sep 10 15:40:53.868849 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.868829    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:53.871095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.871080    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 15:40:53.873802 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.873780    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fzs2q"]
Sep 10 15:40:53.967081 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.967053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpqhk\" (UniqueName: \"kubernetes.io/projected/cb35f097-f180-411a-80a1-771accc542e8-kube-api-access-jpqhk\") pod \"authorino-6d4dfc649f-fzs2q\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") " pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:53.967081 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.967088    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb35f097-f180-411a-80a1-771accc542e8-tls-cert\") pod \"authorino-6d4dfc649f-fzs2q\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") " pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:53.967298 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.967114    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr2kn\" (UniqueName: \"kubernetes.io/projected/b4a58c25-d6a8-4178-9930-2633f4eb65de-kube-api-access-wr2kn\") pod \"authorino-557f876f69-ltw8w\" (UID: \"b4a58c25-d6a8-4178-9930-2633f4eb65de\") " pod="kuadrant-system/authorino-557f876f69-ltw8w"
Sep 10 15:40:53.974643 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:53.974615    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr2kn\" (UniqueName: \"kubernetes.io/projected/b4a58c25-d6a8-4178-9930-2633f4eb65de-kube-api-access-wr2kn\") pod \"authorino-557f876f69-ltw8w\" (UID: \"b4a58c25-d6a8-4178-9930-2633f4eb65de\") " pod="kuadrant-system/authorino-557f876f69-ltw8w"
Sep 10 15:40:54.016302 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.016223    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fzs2q"]
Sep 10 15:40:54.016504 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:40:54.016486    2577 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-jpqhk tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-fzs2q" podUID="cb35f097-f180-411a-80a1-771accc542e8"
Sep 10 15:40:54.043694 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.043659    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-558594464d-c9662"]
Sep 10 15:40:54.047992 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.047975    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.055090 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.055057    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-558594464d-c9662"]
Sep 10 15:40:54.068186 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.068163    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpqhk\" (UniqueName: \"kubernetes.io/projected/cb35f097-f180-411a-80a1-771accc542e8-kube-api-access-jpqhk\") pod \"authorino-6d4dfc649f-fzs2q\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") " pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:54.068299 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.068195    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb35f097-f180-411a-80a1-771accc542e8-tls-cert\") pod \"authorino-6d4dfc649f-fzs2q\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") " pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:54.070776 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.070722    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb35f097-f180-411a-80a1-771accc542e8-tls-cert\") pod \"authorino-6d4dfc649f-fzs2q\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") " pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:54.075385 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.075357    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpqhk\" (UniqueName: \"kubernetes.io/projected/cb35f097-f180-411a-80a1-771accc542e8-kube-api-access-jpqhk\") pod \"authorino-6d4dfc649f-fzs2q\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") " pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:54.169506 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.169469    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn9c5\" (UniqueName: \"kubernetes.io/projected/593e8a62-83fc-438a-8dc3-972af4afffd6-kube-api-access-zn9c5\") pod \"authorino-558594464d-c9662\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") " pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.169672 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.169548    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593e8a62-83fc-438a-8dc3-972af4afffd6-tls-cert\") pod \"authorino-558594464d-c9662\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") " pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.217984 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.217951    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:54.218147 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.217959    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltw8w"
Sep 10 15:40:54.226706 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.226674    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:54.230596 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.230574    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltw8w"
Sep 10 15:40:54.270885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.270791    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpqhk\" (UniqueName: \"kubernetes.io/projected/cb35f097-f180-411a-80a1-771accc542e8-kube-api-access-jpqhk\") pod \"cb35f097-f180-411a-80a1-771accc542e8\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") "
Sep 10 15:40:54.270885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.270871    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb35f097-f180-411a-80a1-771accc542e8-tls-cert\") pod \"cb35f097-f180-411a-80a1-771accc542e8\" (UID: \"cb35f097-f180-411a-80a1-771accc542e8\") "
Sep 10 15:40:54.271081 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.270906    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr2kn\" (UniqueName: \"kubernetes.io/projected/b4a58c25-d6a8-4178-9930-2633f4eb65de-kube-api-access-wr2kn\") pod \"b4a58c25-d6a8-4178-9930-2633f4eb65de\" (UID: \"b4a58c25-d6a8-4178-9930-2633f4eb65de\") "
Sep 10 15:40:54.271081 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.271011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn9c5\" (UniqueName: \"kubernetes.io/projected/593e8a62-83fc-438a-8dc3-972af4afffd6-kube-api-access-zn9c5\") pod \"authorino-558594464d-c9662\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") " pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.271170 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.271103    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593e8a62-83fc-438a-8dc3-972af4afffd6-tls-cert\") pod \"authorino-558594464d-c9662\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") " pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.273207 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.273173    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb35f097-f180-411a-80a1-771accc542e8-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "cb35f097-f180-411a-80a1-771accc542e8" (UID: "cb35f097-f180-411a-80a1-771accc542e8"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:40:54.273325 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.273217    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb35f097-f180-411a-80a1-771accc542e8-kube-api-access-jpqhk" (OuterVolumeSpecName: "kube-api-access-jpqhk") pod "cb35f097-f180-411a-80a1-771accc542e8" (UID: "cb35f097-f180-411a-80a1-771accc542e8"). InnerVolumeSpecName "kube-api-access-jpqhk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:54.273325 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.273269    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4a58c25-d6a8-4178-9930-2633f4eb65de-kube-api-access-wr2kn" (OuterVolumeSpecName: "kube-api-access-wr2kn") pod "b4a58c25-d6a8-4178-9930-2633f4eb65de" (UID: "b4a58c25-d6a8-4178-9930-2633f4eb65de"). InnerVolumeSpecName "kube-api-access-wr2kn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:54.273721 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.273705    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593e8a62-83fc-438a-8dc3-972af4afffd6-tls-cert\") pod \"authorino-558594464d-c9662\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") " pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.278845 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.278822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn9c5\" (UniqueName: \"kubernetes.io/projected/593e8a62-83fc-438a-8dc3-972af4afffd6-kube-api-access-zn9c5\") pod \"authorino-558594464d-c9662\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") " pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.358274 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.358231    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:40:54.372203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.372180    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpqhk\" (UniqueName: \"kubernetes.io/projected/cb35f097-f180-411a-80a1-771accc542e8-kube-api-access-jpqhk\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:54.372309 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.372204    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/cb35f097-f180-411a-80a1-771accc542e8-tls-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:54.372309 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.372216    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr2kn\" (UniqueName: \"kubernetes.io/projected/b4a58c25-d6a8-4178-9930-2633f4eb65de-kube-api-access-wr2kn\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:54.629326 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:54.629301    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-558594464d-c9662"]
Sep 10 15:40:54.631396 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:40:54.631369    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod593e8a62_83fc_438a_8dc3_972af4afffd6.slice/crio-10961b55454671ccb5b77e5d709b29d24efde9a4770cb6da5e9826e4afebefef WatchSource:0}: Error finding container 10961b55454671ccb5b77e5d709b29d24efde9a4770cb6da5e9826e4afebefef: Status 404 returned error can't find the container with id 10961b55454671ccb5b77e5d709b29d24efde9a4770cb6da5e9826e4afebefef
Sep 10 15:40:55.225249 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.225149    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558594464d-c9662" event={"ID":"593e8a62-83fc-438a-8dc3-972af4afffd6","Type":"ContainerStarted","Data":"d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99"}
Sep 10 15:40:55.225249 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.225195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-ltw8w"
Sep 10 15:40:55.225249 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.225206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558594464d-c9662" event={"ID":"593e8a62-83fc-438a-8dc3-972af4afffd6","Type":"ContainerStarted","Data":"10961b55454671ccb5b77e5d709b29d24efde9a4770cb6da5e9826e4afebefef"}
Sep 10 15:40:55.225741 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.225449    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-fzs2q"
Sep 10 15:40:55.257061 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.257033    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fzs2q"]
Sep 10 15:40:55.263692 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.263663    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-fzs2q"]
Sep 10 15:40:55.286236 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.286211    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltw8w"]
Sep 10 15:40:55.290409 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.290375    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-ltw8w"]
Sep 10 15:40:55.308965 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.308922    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-558594464d-c9662" podStartSLOduration=0.96953498 podStartE2EDuration="1.308908505s" totalImagesPullingTime="339.373525ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:40:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:40:55.308131171 +0000 UTC m=+498.922258075" watchObservedRunningTime="2026-09-10 15:40:55.308908505 +0000 UTC m=+498.923035409"
Sep 10 15:40:55.335220 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.335190    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mcmrb"]
Sep 10 15:40:55.335418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.335369    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-mcmrb" podUID="0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f" containerName="authorino" containerID="cri-o://2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693" gracePeriod=30
Sep 10 15:40:55.579035 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.579013    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mcmrb"
Sep 10 15:40:55.683628 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.683597    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghfqg\" (UniqueName: \"kubernetes.io/projected/0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f-kube-api-access-ghfqg\") pod \"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f\" (UID: \"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f\") "
Sep 10 15:40:55.685785 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.685744    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f-kube-api-access-ghfqg" (OuterVolumeSpecName: "kube-api-access-ghfqg") pod "0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f" (UID: "0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f"). InnerVolumeSpecName "kube-api-access-ghfqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:40:55.784141 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:55.784056    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghfqg\" (UniqueName: \"kubernetes.io/projected/0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f-kube-api-access-ghfqg\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:40:56.231185 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.231151    2577 generic.go:356] "Generic (PLEG): container finished" podID="0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f" containerID="2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693" exitCode=0
Sep 10 15:40:56.231604 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.231196    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-mcmrb"
Sep 10 15:40:56.231604 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.231228    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mcmrb" event={"ID":"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f","Type":"ContainerDied","Data":"2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693"}
Sep 10 15:40:56.231604 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.231263    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-mcmrb" event={"ID":"0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f","Type":"ContainerDied","Data":"fd610ae9b6b917327e0cf7106ba60a391c36d891c6002b928d0ed4930daa4b9f"}
Sep 10 15:40:56.231604 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.231280    2577 scope.go:117] "RemoveContainer" containerID="2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693"
Sep 10 15:40:56.240398 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.240380    2577 scope.go:117] "RemoveContainer" containerID="2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693"
Sep 10 15:40:56.240665 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:40:56.240647    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693\": container with ID starting with 2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693 not found: ID does not exist" containerID="2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693"
Sep 10 15:40:56.240731 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.240672    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693"} err="failed to get container status \"2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693\": rpc error: code = NotFound desc = could not find container \"2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693\": container with ID starting with 2efe64dc9ac55745be9573eae63691dd9a4f689809c469a222a54effdf596693 not found: ID does not exist"
Sep 10 15:40:56.251292 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.251270    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mcmrb"]
Sep 10 15:40:56.253952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.253932    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-mcmrb"]
Sep 10 15:40:56.913183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.913139    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f" path="/var/lib/kubelet/pods/0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f/volumes"
Sep 10 15:40:56.913466 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.913454    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4a58c25-d6a8-4178-9930-2633f4eb65de" path="/var/lib/kubelet/pods/b4a58c25-d6a8-4178-9930-2633f4eb65de/volumes"
Sep 10 15:40:56.913668 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:40:56.913657    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb35f097-f180-411a-80a1-771accc542e8" path="/var/lib/kubelet/pods/cb35f097-f180-411a-80a1-771accc542e8/volumes"
Sep 10 15:41:00.253160 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.253130    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-649f667dbc-gjn88"]
Sep 10 15:41:00.253576 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.253562    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f" containerName="authorino"
Sep 10 15:41:00.253621 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.253580    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f" containerName="authorino"
Sep 10 15:41:00.253678 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.253668    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a04acc2-f22f-4cc5-9d95-ac0df80c7c6f" containerName="authorino"
Sep 10 15:41:00.256900 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.256879    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.259592 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.259559    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 15:41:00.259592 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.259585    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w22h2\""
Sep 10 15:41:00.259786 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.259599    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 15:41:00.259786 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.259622    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 15:41:00.268661 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.268637    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-649f667dbc-gjn88"]
Sep 10 15:41:00.321933 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.321904    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp68v\" (UniqueName: \"kubernetes.io/projected/7a2dda57-5698-4023-81f3-f294afbaa5b5-kube-api-access-zp68v\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.322103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.321951    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-metrics-certs\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.322103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.322015    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.423389 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.423355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.423586 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.423421    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp68v\" (UniqueName: \"kubernetes.io/projected/7a2dda57-5698-4023-81f3-f294afbaa5b5-kube-api-access-zp68v\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.423586 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.423479    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-metrics-certs\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.423712 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:41:00.423593    2577 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 10 15:41:00.423712 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:41:00.423665    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert podName:7a2dda57-5698-4023-81f3-f294afbaa5b5 nodeName:}" failed. No retries permitted until 2026-09-10 15:41:00.923645746 +0000 UTC m=+504.537772645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert") pod "maas-controller-649f667dbc-gjn88" (UID: "7a2dda57-5698-4023-81f3-f294afbaa5b5") : secret "maas-controller-webhook-cert" not found
Sep 10 15:41:00.426164 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.426143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-metrics-certs\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.436960 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.436941    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp68v\" (UniqueName: \"kubernetes.io/projected/7a2dda57-5698-4023-81f3-f294afbaa5b5-kube-api-access-zp68v\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.928376 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.928346    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:00.930805 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:00.930776    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert\") pod \"maas-controller-649f667dbc-gjn88\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") " pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:01.167234 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:01.167197    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:01.439679 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:01.439655    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-649f667dbc-gjn88"]
Sep 10 15:41:01.440967 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:41:01.440942    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a2dda57_5698_4023_81f3_f294afbaa5b5.slice/crio-d9b70280c63f0473cf305f05ac6077bf0eb6424c889df5bdf37ea98955a79d1f WatchSource:0}: Error finding container d9b70280c63f0473cf305f05ac6077bf0eb6424c889df5bdf37ea98955a79d1f: Status 404 returned error can't find the container with id d9b70280c63f0473cf305f05ac6077bf0eb6424c889df5bdf37ea98955a79d1f
Sep 10 15:41:02.268975 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:02.268933    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-649f667dbc-gjn88" event={"ID":"7a2dda57-5698-4023-81f3-f294afbaa5b5","Type":"ContainerStarted","Data":"d9b70280c63f0473cf305f05ac6077bf0eb6424c889df5bdf37ea98955a79d1f"}
Sep 10 15:41:04.282813 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:04.282773    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-649f667dbc-gjn88" event={"ID":"7a2dda57-5698-4023-81f3-f294afbaa5b5","Type":"ContainerStarted","Data":"127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958"}
Sep 10 15:41:04.283231 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:04.282841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:04.304908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:04.304851    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-649f667dbc-gjn88" podStartSLOduration=2.1120614030000002 podStartE2EDuration="4.304821236s" totalImagesPullingTime="2.192759833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:04.30153476 +0000 UTC m=+507.915661677" watchObservedRunningTime="2026-09-10 15:41:04.304821236 +0000 UTC m=+507.918948141"
Sep 10 15:41:15.290198 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.290168    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:15.516223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.516196    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-649f667dbc-gjn88"]
Sep 10 15:41:15.516425 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.516405    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-649f667dbc-gjn88" podUID="7a2dda57-5698-4023-81f3-f294afbaa5b5" containerName="manager" containerID="cri-o://127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958" gracePeriod=30
Sep 10 15:41:15.782337 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.782311    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:15.860721 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.860697    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zp68v\" (UniqueName: \"kubernetes.io/projected/7a2dda57-5698-4023-81f3-f294afbaa5b5-kube-api-access-zp68v\") pod \"7a2dda57-5698-4023-81f3-f294afbaa5b5\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") "
Sep 10 15:41:15.860903 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.860727    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-metrics-certs\") pod \"7a2dda57-5698-4023-81f3-f294afbaa5b5\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") "
Sep 10 15:41:15.860975 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.860899    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert\") pod \"7a2dda57-5698-4023-81f3-f294afbaa5b5\" (UID: \"7a2dda57-5698-4023-81f3-f294afbaa5b5\") "
Sep 10 15:41:15.863074 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.863046    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7a2dda57-5698-4023-81f3-f294afbaa5b5" (UID: "7a2dda57-5698-4023-81f3-f294afbaa5b5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:41:15.863183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.863070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7a2dda57-5698-4023-81f3-f294afbaa5b5" (UID: "7a2dda57-5698-4023-81f3-f294afbaa5b5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:41:15.863183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.863070    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a2dda57-5698-4023-81f3-f294afbaa5b5-kube-api-access-zp68v" (OuterVolumeSpecName: "kube-api-access-zp68v") pod "7a2dda57-5698-4023-81f3-f294afbaa5b5" (UID: "7a2dda57-5698-4023-81f3-f294afbaa5b5"). InnerVolumeSpecName "kube-api-access-zp68v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:41:15.962121 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.962088    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zp68v\" (UniqueName: \"kubernetes.io/projected/7a2dda57-5698-4023-81f3-f294afbaa5b5-kube-api-access-zp68v\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.962121 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.962115    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:41:15.962121 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:15.962124    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7a2dda57-5698-4023-81f3-f294afbaa5b5-webhook-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:41:16.342813 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.342783    2577 generic.go:356] "Generic (PLEG): container finished" podID="7a2dda57-5698-4023-81f3-f294afbaa5b5" containerID="127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958" exitCode=0
Sep 10 15:41:16.343280 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.342843    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-649f667dbc-gjn88"
Sep 10 15:41:16.343280 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.342872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-649f667dbc-gjn88" event={"ID":"7a2dda57-5698-4023-81f3-f294afbaa5b5","Type":"ContainerDied","Data":"127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958"}
Sep 10 15:41:16.343280 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.342940    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-649f667dbc-gjn88" event={"ID":"7a2dda57-5698-4023-81f3-f294afbaa5b5","Type":"ContainerDied","Data":"d9b70280c63f0473cf305f05ac6077bf0eb6424c889df5bdf37ea98955a79d1f"}
Sep 10 15:41:16.343280 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.342976    2577 scope.go:117] "RemoveContainer" containerID="127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958"
Sep 10 15:41:16.352150 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.352134    2577 scope.go:117] "RemoveContainer" containerID="127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958"
Sep 10 15:41:16.352424 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:41:16.352407    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958\": container with ID starting with 127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958 not found: ID does not exist" containerID="127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958"
Sep 10 15:41:16.352466 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.352434    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958"} err="failed to get container status \"127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958\": rpc error: code = NotFound desc = could not find container \"127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958\": container with ID starting with 127cc0a11faccab03b1c321fd0fd1d95aacc631979bfec5659dcc82d0dd7d958 not found: ID does not exist"
Sep 10 15:41:16.363375 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.363354    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-649f667dbc-gjn88"]
Sep 10 15:41:16.367512 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.367484    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-649f667dbc-gjn88"]
Sep 10 15:41:16.907566 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:16.907537    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a2dda57-5698-4023-81f3-f294afbaa5b5" path="/var/lib/kubelet/pods/7a2dda57-5698-4023-81f3-f294afbaa5b5/volumes"
Sep 10 15:41:40.108206 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.108173    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-699674684b-pt4jc"]
Sep 10 15:41:40.108657 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.108642    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a2dda57-5698-4023-81f3-f294afbaa5b5" containerName="manager"
Sep 10 15:41:40.108700 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.108660    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a2dda57-5698-4023-81f3-f294afbaa5b5" containerName="manager"
Sep 10 15:41:40.108746 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.108736    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a2dda57-5698-4023-81f3-f294afbaa5b5" containerName="manager"
Sep 10 15:41:40.113890 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.113869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.117309 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.117289    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 15:41:40.117442 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.117312    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ql4pw\""
Sep 10 15:41:40.117442 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.117318    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 15:41:40.121934 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.121913    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699674684b-pt4jc"]
Sep 10 15:41:40.130946 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.130925    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgcnv\" (UniqueName: \"kubernetes.io/projected/b251bb68-9eda-487e-bb07-8d0ee793ff4f-kube-api-access-zgcnv\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.131040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.130958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251bb68-9eda-487e-bb07-8d0ee793ff4f-openshift-service-ca\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.131040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.130985    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-metrics-certs\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.131119 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.131053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-maas-api-tls\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.231608 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.231582    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-maas-api-tls\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.231739 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.231632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgcnv\" (UniqueName: \"kubernetes.io/projected/b251bb68-9eda-487e-bb07-8d0ee793ff4f-kube-api-access-zgcnv\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.231739 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.231665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251bb68-9eda-487e-bb07-8d0ee793ff4f-openshift-service-ca\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.231739 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.231706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-metrics-certs\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.232415 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.232338    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251bb68-9eda-487e-bb07-8d0ee793ff4f-openshift-service-ca\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.234357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.234333    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-metrics-certs\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.234357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.234349    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-maas-api-tls\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.240099 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.240073    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgcnv\" (UniqueName: \"kubernetes.io/projected/b251bb68-9eda-487e-bb07-8d0ee793ff4f-kube-api-access-zgcnv\") pod \"maas-api-699674684b-pt4jc\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") " pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.426400 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.426328    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:40.490023 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.489993    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"]
Sep 10 15:41:40.494737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.494712    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:40.498384 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.498125    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 10 15:41:40.498384 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.498169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-6cqbb\""
Sep 10 15:41:40.506377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.506347    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"]
Sep 10 15:41:40.534400 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.534377    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwfmz\" (UniqueName: \"kubernetes.io/projected/92a2bd37-c4e6-4592-b933-db111b0fb2e3-kube-api-access-wwfmz\") pod \"payload-pre-processing-59fbc47bf8-2zdst\" (UID: \"92a2bd37-c4e6-4592-b933-db111b0fb2e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:40.534526 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.534422    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a2bd37-c4e6-4592-b933-db111b0fb2e3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2zdst\" (UID: \"92a2bd37-c4e6-4592-b933-db111b0fb2e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:40.635040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.635002    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwfmz\" (UniqueName: \"kubernetes.io/projected/92a2bd37-c4e6-4592-b933-db111b0fb2e3-kube-api-access-wwfmz\") pod \"payload-pre-processing-59fbc47bf8-2zdst\" (UID: \"92a2bd37-c4e6-4592-b933-db111b0fb2e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:40.635233 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.635073    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a2bd37-c4e6-4592-b933-db111b0fb2e3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2zdst\" (UID: \"92a2bd37-c4e6-4592-b933-db111b0fb2e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:40.635783 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.635737    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/92a2bd37-c4e6-4592-b933-db111b0fb2e3-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-2zdst\" (UID: \"92a2bd37-c4e6-4592-b933-db111b0fb2e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:40.646047 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.646016    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwfmz\" (UniqueName: \"kubernetes.io/projected/92a2bd37-c4e6-4592-b933-db111b0fb2e3-kube-api-access-wwfmz\") pod \"payload-pre-processing-59fbc47bf8-2zdst\" (UID: \"92a2bd37-c4e6-4592-b933-db111b0fb2e3\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:40.703141 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.703111    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-699674684b-pt4jc"]
Sep 10 15:41:40.704692 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:41:40.704668    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb251bb68_9eda_487e_bb07_8d0ee793ff4f.slice/crio-376c7f1b6becdbca30663f24ff1768f854d8698165b49c8642df8c3a92e0a6b8 WatchSource:0}: Error finding container 376c7f1b6becdbca30663f24ff1768f854d8698165b49c8642df8c3a92e0a6b8: Status 404 returned error can't find the container with id 376c7f1b6becdbca30663f24ff1768f854d8698165b49c8642df8c3a92e0a6b8
Sep 10 15:41:40.807167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:40.807133    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:41.087227 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:41.087199    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"]
Sep 10 15:41:41.089269 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:41:41.089243    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a2bd37_c4e6_4592_b933_db111b0fb2e3.slice/crio-7fb81f14696ba283613c650ebe3d22bd7c608b053e082237927679a4242fbf6c WatchSource:0}: Error finding container 7fb81f14696ba283613c650ebe3d22bd7c608b053e082237927679a4242fbf6c: Status 404 returned error can't find the container with id 7fb81f14696ba283613c650ebe3d22bd7c608b053e082237927679a4242fbf6c
Sep 10 15:41:41.467103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:41.467014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc" event={"ID":"b251bb68-9eda-487e-bb07-8d0ee793ff4f","Type":"ContainerStarted","Data":"376c7f1b6becdbca30663f24ff1768f854d8698165b49c8642df8c3a92e0a6b8"}
Sep 10 15:41:41.468414 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:41.468383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst" event={"ID":"92a2bd37-c4e6-4592-b933-db111b0fb2e3","Type":"ContainerStarted","Data":"7fb81f14696ba283613c650ebe3d22bd7c608b053e082237927679a4242fbf6c"}
Sep 10 15:41:43.479313 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:43.479278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst" event={"ID":"92a2bd37-c4e6-4592-b933-db111b0fb2e3","Type":"ContainerStarted","Data":"9b090c618e6ff36cd48aca5b61436b1d1ce27e5031dc10f65a705ca9faf98398"}
Sep 10 15:41:43.479716 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:43.479344    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:43.480877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:43.480854    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc" event={"ID":"b251bb68-9eda-487e-bb07-8d0ee793ff4f","Type":"ContainerStarted","Data":"8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0"}
Sep 10 15:41:43.481001 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:43.480975    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:43.496694 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:43.496653    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst" podStartSLOduration=1.74245026 podStartE2EDuration="3.496639341s" totalImagesPullingTime="1.754189081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:43.494620119 +0000 UTC m=+547.108747024" watchObservedRunningTime="2026-09-10 15:41:43.496639341 +0000 UTC m=+547.110766244"
Sep 10 15:41:43.512195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:43.512152    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc" podStartSLOduration=1.422458108 podStartE2EDuration="3.512134791s" totalImagesPullingTime="2.089676683s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:43.511581487 +0000 UTC m=+547.125708394" watchObservedRunningTime="2026-09-10 15:41:43.512134791 +0000 UTC m=+547.126261696"
Sep 10 15:41:49.491540 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:49.491512    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 15:41:54.488491 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:54.488463    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-2zdst"
Sep 10 15:41:56.335334 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.335261    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"]
Sep 10 15:41:56.340254 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.340226    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.342829 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.342808    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 15:41:56.343662 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.343635    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 10 15:41:56.343785 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.343674    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ntw7b\""
Sep 10 15:41:56.343785 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.343678    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 15:41:56.350071 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.350047    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"]
Sep 10 15:41:56.466309 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.466275    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pr5w\" (UniqueName: \"kubernetes.io/projected/8dac9953-e6c6-4281-92aa-26684f111031-kube-api-access-2pr5w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.466496 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.466314    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.466496 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.466354    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dac9953-e6c6-4281-92aa-26684f111031-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.466496 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.466431    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.466496 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.466474    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.466642 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.466512    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.567932 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.567896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568108 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.567941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568108 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.567978    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pr5w\" (UniqueName: \"kubernetes.io/projected/8dac9953-e6c6-4281-92aa-26684f111031-kube-api-access-2pr5w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568108 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.568081    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.568155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dac9953-e6c6-4281-92aa-26684f111031-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.568204    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568385 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.568368    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.568473    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.568552 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.568465    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.570603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.570575    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dac9953-e6c6-4281-92aa-26684f111031-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.570833 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.570814    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dac9953-e6c6-4281-92aa-26684f111031-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.576628 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.576609    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pr5w\" (UniqueName: \"kubernetes.io/projected/8dac9953-e6c6-4281-92aa-26684f111031-kube-api-access-2pr5w\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj\" (UID: \"8dac9953-e6c6-4281-92aa-26684f111031\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.651581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.651488    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:41:56.928814 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:56.928786    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"]
Sep 10 15:41:56.929566 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:41:56.929536    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dac9953_e6c6_4281_92aa_26684f111031.slice/crio-5d434d1db8f54c883217784c91421838feaf6477ed824bd237b2cf45133059aa WatchSource:0}: Error finding container 5d434d1db8f54c883217784c91421838feaf6477ed824bd237b2cf45133059aa: Status 404 returned error can't find the container with id 5d434d1db8f54c883217784c91421838feaf6477ed824bd237b2cf45133059aa
Sep 10 15:41:57.555627 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:57.555586    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj" event={"ID":"8dac9953-e6c6-4281-92aa-26684f111031","Type":"ContainerStarted","Data":"5d434d1db8f54c883217784c91421838feaf6477ed824bd237b2cf45133059aa"}
Sep 10 15:41:58.562906 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:58.562817    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj" event={"ID":"8dac9953-e6c6-4281-92aa-26684f111031","Type":"ContainerStarted","Data":"0d430953ac68c8e049c64510614d45689f07336f103e5f1acb5abccc0743a3dc"}
Sep 10 15:41:58.585841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:41:58.584817    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj" podStartSLOduration=1.3611326670000001 podStartE2EDuration="2.584794859s" totalImagesPullingTime="1.223662192s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:41:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:41:58.579261283 +0000 UTC m=+562.193388186" watchObservedRunningTime="2026-09-10 15:41:58.584794859 +0000 UTC m=+562.198921765"
Sep 10 15:42:06.652493 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:06.652445    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:42:06.653014 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:06.652556    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:42:06.665167 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:06.665144    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:42:07.618568 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.618542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj"
Sep 10 15:42:07.851668 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.851640    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"]
Sep 10 15:42:07.855175 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.855151    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:07.857623 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.857600    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\""
Sep 10 15:42:07.864019 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.863997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"]
Sep 10 15:42:07.969634 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.969554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:07.969634 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.969586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b00eea2-d0cf-4bc9-9bca-a354c796952f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:07.969634 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.969619    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cktkd\" (UniqueName: \"kubernetes.io/projected/1b00eea2-d0cf-4bc9-9bca-a354c796952f-kube-api-access-cktkd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:07.969858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.969707    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:07.969858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.969743    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:07.969858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:07.969801    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.070899 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.070861    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.070899 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.070901    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.071096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.070922    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.071096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.070958    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.071096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.070975    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b00eea2-d0cf-4bc9-9bca-a354c796952f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.071096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.071005    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cktkd\" (UniqueName: \"kubernetes.io/projected/1b00eea2-d0cf-4bc9-9bca-a354c796952f-kube-api-access-cktkd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.071293 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.071271    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-home\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.071352 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.071323    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-model-cache\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.071433 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.071411    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-tmp-dir\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.073314 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.073294    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b00eea2-d0cf-4bc9-9bca-a354c796952f-dshm\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.073565 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.073550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b00eea2-d0cf-4bc9-9bca-a354c796952f-tls-certs\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.079187 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.079162    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cktkd\" (UniqueName: \"kubernetes.io/projected/1b00eea2-d0cf-4bc9-9bca-a354c796952f-kube-api-access-cktkd\") pod \"e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd\" (UID: \"1b00eea2-d0cf-4bc9-9bca-a354c796952f\") " pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.165595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.165564    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:08.440098 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.440075    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"]
Sep 10 15:42:08.441906 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:42:08.441880    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b00eea2_d0cf_4bc9_9bca_a354c796952f.slice/crio-5af99f35c8f59e8065fefd9d4a347925cae1260323cf60d56710b5bff3865080 WatchSource:0}: Error finding container 5af99f35c8f59e8065fefd9d4a347925cae1260323cf60d56710b5bff3865080: Status 404 returned error can't find the container with id 5af99f35c8f59e8065fefd9d4a347925cae1260323cf60d56710b5bff3865080
Sep 10 15:42:08.614096 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:08.614059    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd" event={"ID":"1b00eea2-d0cf-4bc9-9bca-a354c796952f","Type":"ContainerStarted","Data":"5af99f35c8f59e8065fefd9d4a347925cae1260323cf60d56710b5bff3865080"}
Sep 10 15:42:09.622309 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:09.622276    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd" event={"ID":"1b00eea2-d0cf-4bc9-9bca-a354c796952f","Type":"ContainerStarted","Data":"b9d92ef789a66e50326d998378c5508eccc12722c8a9d42af61bb0db3ee9a2b9"}
Sep 10 15:42:09.639512 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:09.639454    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd" podStartSLOduration=2.429060286 podStartE2EDuration="2.639437834s" totalImagesPullingTime="210.377548ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:09.638063727 +0000 UTC m=+573.252190671" watchObservedRunningTime="2026-09-10 15:42:09.639437834 +0000 UTC m=+573.253564738"
Sep 10 15:42:18.166782 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:18.166717    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:18.167430 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:18.166832    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:18.179828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:18.179805    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:42:18.683464 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:42:18.683432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd"
Sep 10 15:43:57.440156 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.440118    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75fb6787d5-2zngj"]
Sep 10 15:43:57.442655 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.442639    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.445238 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.445214    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 10 15:43:57.445368 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.445256    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 10 15:43:57.446297 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.446271    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 10 15:43:57.446297 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.446293    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-jdlj5\""
Sep 10 15:43:57.446459 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.446314    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 10 15:43:57.446459 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.446344    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 10 15:43:57.446459 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.446415    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 10 15:43:57.446459 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.446429    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 10 15:43:57.451161 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.451142    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 10 15:43:57.455554 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.455530    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fb6787d5-2zngj"]
Sep 10 15:43:57.534332 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.534303    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-config\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.534332 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.534333    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-service-ca\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.534543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.534352    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-trusted-ca-bundle\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.534543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.534371    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-serving-cert\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.534543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.534414    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-oauth-config\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.534543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.534461    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dq786\" (UniqueName: \"kubernetes.io/projected/f0db9eda-9772-4212-8af5-d1d89a9b7c96-kube-api-access-dq786\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.534543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.534506    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-oauth-serving-cert\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.635370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.635339    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-config\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.635370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.635372    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-service-ca\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.635589 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.635389    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-trusted-ca-bundle\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.635589 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.635408    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-serving-cert\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.635589 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.635422    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-oauth-config\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.635589 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.635465    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dq786\" (UniqueName: \"kubernetes.io/projected/f0db9eda-9772-4212-8af5-d1d89a9b7c96-kube-api-access-dq786\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.635589 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.635526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-oauth-serving-cert\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.636223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.636179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-service-ca\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.636223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.636179    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-config\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.636406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.636267    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-trusted-ca-bundle\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.636583 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.636562    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/f0db9eda-9772-4212-8af5-d1d89a9b7c96-oauth-serving-cert\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.638068 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.638047    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-serving-cert\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.638171 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.638075    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/f0db9eda-9772-4212-8af5-d1d89a9b7c96-console-oauth-config\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.643209 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.643192    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dq786\" (UniqueName: \"kubernetes.io/projected/f0db9eda-9772-4212-8af5-d1d89a9b7c96-kube-api-access-dq786\") pod \"console-75fb6787d5-2zngj\" (UID: \"f0db9eda-9772-4212-8af5-d1d89a9b7c96\") " pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:57.752707 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:57.752636    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:43:58.024223 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:58.024197    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75fb6787d5-2zngj"]
Sep 10 15:43:58.026486 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:43:58.026460    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0db9eda_9772_4212_8af5_d1d89a9b7c96.slice/crio-bf2cdaceb88350022052e06fd495a7c852163462eadc30d857051a512c4afbf5 WatchSource:0}: Error finding container bf2cdaceb88350022052e06fd495a7c852163462eadc30d857051a512c4afbf5: Status 404 returned error can't find the container with id bf2cdaceb88350022052e06fd495a7c852163462eadc30d857051a512c4afbf5
Sep 10 15:43:58.175745 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:58.175708    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fb6787d5-2zngj" event={"ID":"f0db9eda-9772-4212-8af5-d1d89a9b7c96","Type":"ContainerStarted","Data":"3f1f4893b06acfa6c5cc34b4979f1fa2fb2c9bf57d173cc354636460fb385500"}
Sep 10 15:43:58.175935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:58.175797    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75fb6787d5-2zngj" event={"ID":"f0db9eda-9772-4212-8af5-d1d89a9b7c96","Type":"ContainerStarted","Data":"bf2cdaceb88350022052e06fd495a7c852163462eadc30d857051a512c4afbf5"}
Sep 10 15:43:58.193296 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:43:58.193243    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75fb6787d5-2zngj" podStartSLOduration=1.193227603 podStartE2EDuration="1.193227603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:58.192689311 +0000 UTC m=+681.806816217" watchObservedRunningTime="2026-09-10 15:43:58.193227603 +0000 UTC m=+681.807354508"
Sep 10 15:44:07.753835 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:07.753792    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:44:07.754328 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:07.753862    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:44:07.758761 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:07.758731    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:44:08.243648 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:08.243621    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75fb6787d5-2zngj"
Sep 10 15:44:35.833504 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:35.833427    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7676795785-b4j99"]
Sep 10 15:44:35.840064 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:35.836685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:35.860422 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:35.860390    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7676795785-b4j99"]
Sep 10 15:44:35.960311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:35.960278    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxdv2\" (UniqueName: \"kubernetes.io/projected/99bd3d28-a934-479d-a7d4-0a9abdd179fc-kube-api-access-gxdv2\") pod \"authorino-7676795785-b4j99\" (UID: \"99bd3d28-a934-479d-a7d4-0a9abdd179fc\") " pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:35.960449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:35.960340    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99bd3d28-a934-479d-a7d4-0a9abdd179fc-tls-cert\") pod \"authorino-7676795785-b4j99\" (UID: \"99bd3d28-a934-479d-a7d4-0a9abdd179fc\") " pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:36.061542 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:36.061503    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxdv2\" (UniqueName: \"kubernetes.io/projected/99bd3d28-a934-479d-a7d4-0a9abdd179fc-kube-api-access-gxdv2\") pod \"authorino-7676795785-b4j99\" (UID: \"99bd3d28-a934-479d-a7d4-0a9abdd179fc\") " pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:36.061696 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:36.061564    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99bd3d28-a934-479d-a7d4-0a9abdd179fc-tls-cert\") pod \"authorino-7676795785-b4j99\" (UID: \"99bd3d28-a934-479d-a7d4-0a9abdd179fc\") " pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:36.064074 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:36.064056    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/99bd3d28-a934-479d-a7d4-0a9abdd179fc-tls-cert\") pod \"authorino-7676795785-b4j99\" (UID: \"99bd3d28-a934-479d-a7d4-0a9abdd179fc\") " pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:36.069238 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:36.069219    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxdv2\" (UniqueName: \"kubernetes.io/projected/99bd3d28-a934-479d-a7d4-0a9abdd179fc-kube-api-access-gxdv2\") pod \"authorino-7676795785-b4j99\" (UID: \"99bd3d28-a934-479d-a7d4-0a9abdd179fc\") " pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:36.149645 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:36.149586    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7676795785-b4j99"
Sep 10 15:44:36.417581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:36.417560    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7676795785-b4j99"]
Sep 10 15:44:36.419665 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:44:36.419639    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod99bd3d28_a934_479d_a7d4_0a9abdd179fc.slice/crio-1f3307e5a1daa3643bbfd6b6bd45327e40e2dc37af79ea395926fbabb887fcc5 WatchSource:0}: Error finding container 1f3307e5a1daa3643bbfd6b6bd45327e40e2dc37af79ea395926fbabb887fcc5: Status 404 returned error can't find the container with id 1f3307e5a1daa3643bbfd6b6bd45327e40e2dc37af79ea395926fbabb887fcc5
Sep 10 15:44:36.420967 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:36.420947    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:44:37.389885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.389061    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7676795785-b4j99" event={"ID":"99bd3d28-a934-479d-a7d4-0a9abdd179fc","Type":"ContainerStarted","Data":"7f5ebaaf5a47214917a6d9a4f9b36470da6138b5e65a8abb20e039998f818229"}
Sep 10 15:44:37.389885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.389129    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7676795785-b4j99" event={"ID":"99bd3d28-a934-479d-a7d4-0a9abdd179fc","Type":"ContainerStarted","Data":"1f3307e5a1daa3643bbfd6b6bd45327e40e2dc37af79ea395926fbabb887fcc5"}
Sep 10 15:44:37.408146 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.408085    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7676795785-b4j99" podStartSLOduration=1.753617064 podStartE2EDuration="2.408063702s" totalImagesPullingTime="654.446638ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:37.407315777 +0000 UTC m=+721.021442685" watchObservedRunningTime="2026-09-10 15:44:37.408063702 +0000 UTC m=+721.022190606"
Sep 10 15:44:37.435497 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.435366    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-558594464d-c9662"]
Sep 10 15:44:37.435704 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.435620    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-558594464d-c9662" podUID="593e8a62-83fc-438a-8dc3-972af4afffd6" containerName="authorino" containerID="cri-o://d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99" gracePeriod=30
Sep 10 15:44:37.704592 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.704567    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:44:37.781258 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.781227    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zn9c5\" (UniqueName: \"kubernetes.io/projected/593e8a62-83fc-438a-8dc3-972af4afffd6-kube-api-access-zn9c5\") pod \"593e8a62-83fc-438a-8dc3-972af4afffd6\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") "
Sep 10 15:44:37.781449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.781282    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593e8a62-83fc-438a-8dc3-972af4afffd6-tls-cert\") pod \"593e8a62-83fc-438a-8dc3-972af4afffd6\" (UID: \"593e8a62-83fc-438a-8dc3-972af4afffd6\") "
Sep 10 15:44:37.791772 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.787857    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/593e8a62-83fc-438a-8dc3-972af4afffd6-kube-api-access-zn9c5" (OuterVolumeSpecName: "kube-api-access-zn9c5") pod "593e8a62-83fc-438a-8dc3-972af4afffd6" (UID: "593e8a62-83fc-438a-8dc3-972af4afffd6"). InnerVolumeSpecName "kube-api-access-zn9c5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:37.803987 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.803945    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/593e8a62-83fc-438a-8dc3-972af4afffd6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "593e8a62-83fc-438a-8dc3-972af4afffd6" (UID: "593e8a62-83fc-438a-8dc3-972af4afffd6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:37.882477 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.882446    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn9c5\" (UniqueName: \"kubernetes.io/projected/593e8a62-83fc-438a-8dc3-972af4afffd6-kube-api-access-zn9c5\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:44:37.882477 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:37.882478    2577 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/593e8a62-83fc-438a-8dc3-972af4afffd6-tls-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:44:38.396347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.396256    2577 generic.go:356] "Generic (PLEG): container finished" podID="593e8a62-83fc-438a-8dc3-972af4afffd6" containerID="d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99" exitCode=0
Sep 10 15:44:38.396347 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.396309    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-558594464d-c9662"
Sep 10 15:44:38.396876 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.396348    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558594464d-c9662" event={"ID":"593e8a62-83fc-438a-8dc3-972af4afffd6","Type":"ContainerDied","Data":"d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99"}
Sep 10 15:44:38.396876 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.396388    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-558594464d-c9662" event={"ID":"593e8a62-83fc-438a-8dc3-972af4afffd6","Type":"ContainerDied","Data":"10961b55454671ccb5b77e5d709b29d24efde9a4770cb6da5e9826e4afebefef"}
Sep 10 15:44:38.396876 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.396409    2577 scope.go:117] "RemoveContainer" containerID="d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99"
Sep 10 15:44:38.406377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.406359    2577 scope.go:117] "RemoveContainer" containerID="d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99"
Sep 10 15:44:38.406687 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:44:38.406668    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99\": container with ID starting with d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99 not found: ID does not exist" containerID="d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99"
Sep 10 15:44:38.406766 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.406701    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99"} err="failed to get container status \"d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99\": rpc error: code = NotFound desc = could not find container \"d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99\": container with ID starting with d806caf980ab767473a1ba0b66b04d5d3fdbb60049b14f5a4a691f6979621c99 not found: ID does not exist"
Sep 10 15:44:38.420880 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.420855    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-558594464d-c9662"]
Sep 10 15:44:38.425298 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.425273    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-558594464d-c9662"]
Sep 10 15:44:38.908451 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:38.908416    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="593e8a62-83fc-438a-8dc3-972af4afffd6" path="/var/lib/kubelet/pods/593e8a62-83fc-438a-8dc3-972af4afffd6/volumes"
Sep 10 15:44:59.597612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.596414    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"]
Sep 10 15:44:59.597612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.596960    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="593e8a62-83fc-438a-8dc3-972af4afffd6" containerName="authorino"
Sep 10 15:44:59.597612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.596978    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="593e8a62-83fc-438a-8dc3-972af4afffd6" containerName="authorino"
Sep 10 15:44:59.597612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.597071    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="593e8a62-83fc-438a-8dc3-972af4afffd6" containerName="authorino"
Sep 10 15:44:59.601895 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.601869    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.604280 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.604253    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-80d47c-serving-cert\""
Sep 10 15:44:59.613154 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.613130    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"]
Sep 10 15:44:59.662535 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.662511    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-metrics-certs\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.662685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.662549    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd823d89-2ba7-4753-bbd0-efb19a18d017-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.662685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.662571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rg4vj\" (UniqueName: \"kubernetes.io/projected/bd823d89-2ba7-4753-bbd0-efb19a18d017-kube-api-access-rg4vj\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.662777 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.662684    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.763935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.763905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd823d89-2ba7-4753-bbd0-efb19a18d017-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.764106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.763949    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rg4vj\" (UniqueName: \"kubernetes.io/projected/bd823d89-2ba7-4753-bbd0-efb19a18d017-kube-api-access-rg4vj\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.764106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.764011    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.764106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.764047    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-metrics-certs\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.764281 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:44:59.764146    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-serving-cert: secret "maas-api-e2e-inf-a-80d47c-serving-cert" not found
Sep 10 15:44:59.764281 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:44:59.764216    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls podName:bd823d89-2ba7-4753-bbd0-efb19a18d017 nodeName:}" failed. No retries permitted until 2026-09-10 15:45:00.264195852 +0000 UTC m=+743.878322748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls") pod "maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt" (UID: "bd823d89-2ba7-4753-bbd0-efb19a18d017") : secret "maas-api-e2e-inf-a-80d47c-serving-cert" not found
Sep 10 15:44:59.764655 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.764636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd823d89-2ba7-4753-bbd0-efb19a18d017-openshift-service-ca\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.766566 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.766547    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-metrics-certs\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:44:59.775086 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:44:59.775062    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rg4vj\" (UniqueName: \"kubernetes.io/projected/bd823d89-2ba7-4753-bbd0-efb19a18d017-kube-api-access-rg4vj\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:45:00.268245 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.268213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:45:00.270661 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.270640    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls\") pod \"maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:45:00.346267 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.346238    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"]
Sep 10 15:45:00.349614 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.349600    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.352040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.352017    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-80d47c-dockercfg-vk8b7\""
Sep 10 15:45:00.352168 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.352054    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-80d47c\""
Sep 10 15:45:00.361216 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.361187    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"]
Sep 10 15:45:00.370354 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.370331    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"]
Sep 10 15:45:00.376552 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.376529    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.381922 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.381898    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"]
Sep 10 15:45:00.469792 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.469738    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57da81d-642d-4c11-976b-18a7da3f2790-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.469935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.469848    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc4n2\" (UniqueName: \"kubernetes.io/projected/90afc743-ab68-4d63-a204-bc6d17e56966-kube-api-access-rc4n2\") pod \"payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.469935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.469896    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90afc743-ab68-4d63-a204-bc6d17e56966-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.470015 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.469958    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvmmq\" (UniqueName: \"kubernetes.io/projected/b57da81d-642d-4c11-976b-18a7da3f2790-kube-api-access-pvmmq\") pod \"payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.515897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.515871    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:45:00.571468 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.571432    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc4n2\" (UniqueName: \"kubernetes.io/projected/90afc743-ab68-4d63-a204-bc6d17e56966-kube-api-access-rc4n2\") pod \"payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.571600 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.571515    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90afc743-ab68-4d63-a204-bc6d17e56966-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.571600 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.571589    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvmmq\" (UniqueName: \"kubernetes.io/projected/b57da81d-642d-4c11-976b-18a7da3f2790-kube-api-access-pvmmq\") pod \"payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.571719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.571633    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57da81d-642d-4c11-976b-18a7da3f2790-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.572627 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.572598    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57da81d-642d-4c11-976b-18a7da3f2790-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.572731 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.572666    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90afc743-ab68-4d63-a204-bc6d17e56966-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.580477 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.580401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc4n2\" (UniqueName: \"kubernetes.io/projected/90afc743-ab68-4d63-a204-bc6d17e56966-kube-api-access-rc4n2\") pod \"payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.580717 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.580672    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvmmq\" (UniqueName: \"kubernetes.io/projected/b57da81d-642d-4c11-976b-18a7da3f2790-kube-api-access-pvmmq\") pod \"payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") " pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.661304 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.661264    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:00.690779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.689847    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:00.800158 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.800024    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"]
Sep 10 15:45:00.804025 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:00.803999    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd823d89_2ba7_4753_bbd0_efb19a18d017.slice/crio-e92f8c11598a523d22a138a93e7620c4ea3164332ab8ce872a68126a35a1be0f WatchSource:0}: Error finding container e92f8c11598a523d22a138a93e7620c4ea3164332ab8ce872a68126a35a1be0f: Status 404 returned error can't find the container with id e92f8c11598a523d22a138a93e7620c4ea3164332ab8ce872a68126a35a1be0f
Sep 10 15:45:00.941433 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.941404    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"]
Sep 10 15:45:00.943166 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:00.943140    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90afc743_ab68_4d63_a204_bc6d17e56966.slice/crio-6ac8252fc7209fcaecfbd045f1c41642e983b21921d59ef7a7c6e458b82e5d8f WatchSource:0}: Error finding container 6ac8252fc7209fcaecfbd045f1c41642e983b21921d59ef7a7c6e458b82e5d8f: Status 404 returned error can't find the container with id 6ac8252fc7209fcaecfbd045f1c41642e983b21921d59ef7a7c6e458b82e5d8f
Sep 10 15:45:00.974106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:00.974076    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"]
Sep 10 15:45:00.983067 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:00.983035    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb57da81d_642d_4c11_976b_18a7da3f2790.slice/crio-2b5b3a4d569ebaf05e7d6d58989fdb802bac3ad15adcd1ced214bdf9ea3e6050 WatchSource:0}: Error finding container 2b5b3a4d569ebaf05e7d6d58989fdb802bac3ad15adcd1ced214bdf9ea3e6050: Status 404 returned error can't find the container with id 2b5b3a4d569ebaf05e7d6d58989fdb802bac3ad15adcd1ced214bdf9ea3e6050
Sep 10 15:45:01.520981 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.520942    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd" event={"ID":"b57da81d-642d-4c11-976b-18a7da3f2790","Type":"ContainerStarted","Data":"4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173"}
Sep 10 15:45:01.521184 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.521014    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:01.521184 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.521039    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd" event={"ID":"b57da81d-642d-4c11-976b-18a7da3f2790","Type":"ContainerStarted","Data":"2b5b3a4d569ebaf05e7d6d58989fdb802bac3ad15adcd1ced214bdf9ea3e6050"}
Sep 10 15:45:01.522635 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.522607    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff" event={"ID":"90afc743-ab68-4d63-a204-bc6d17e56966","Type":"ContainerStarted","Data":"62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25"}
Sep 10 15:45:01.522787 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.522650    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff" event={"ID":"90afc743-ab68-4d63-a204-bc6d17e56966","Type":"ContainerStarted","Data":"6ac8252fc7209fcaecfbd045f1c41642e983b21921d59ef7a7c6e458b82e5d8f"}
Sep 10 15:45:01.522787 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.522703    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:01.523869 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.523849    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt" event={"ID":"bd823d89-2ba7-4753-bbd0-efb19a18d017","Type":"ContainerStarted","Data":"e92f8c11598a523d22a138a93e7620c4ea3164332ab8ce872a68126a35a1be0f"}
Sep 10 15:45:01.539136 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.539081    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd" podStartSLOduration=1.5390636610000001 podStartE2EDuration="1.539063661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:01.536518134 +0000 UTC m=+745.150645039" watchObservedRunningTime="2026-09-10 15:45:01.539063661 +0000 UTC m=+745.153190563"
Sep 10 15:45:01.555868 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:01.555816    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff" podStartSLOduration=1.555795139 podStartE2EDuration="1.555795139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:01.555645391 +0000 UTC m=+745.169772288" watchObservedRunningTime="2026-09-10 15:45:01.555795139 +0000 UTC m=+745.169922044"
Sep 10 15:45:02.531833 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:02.531799    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt" event={"ID":"bd823d89-2ba7-4753-bbd0-efb19a18d017","Type":"ContainerStarted","Data":"071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f"}
Sep 10 15:45:02.532272 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:02.531901    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:45:02.550669 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:02.550626    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt" podStartSLOduration=2.812358088 podStartE2EDuration="3.550611395s" totalImagesPullingTime="738.253307ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:44:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:02.54884462 +0000 UTC m=+746.162971524" watchObservedRunningTime="2026-09-10 15:45:02.550611395 +0000 UTC m=+746.164738299"
Sep 10 15:45:07.209063 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.209030    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"]
Sep 10 15:45:07.213342 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.213318    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.216079 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.216055    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-7083b0df-serving-cert\""
Sep 10 15:45:07.223362 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.223328    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"]
Sep 10 15:45:07.332264 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.332223    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-metrics-certs\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.332264 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.332264    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d52-32a3-4a1f-a306-87f662db3581-openshift-service-ca\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.332509 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.332390    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.332565 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.332505    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jss7t\" (UniqueName: \"kubernetes.io/projected/e1859d52-32a3-4a1f-a306-87f662db3581-kube-api-access-jss7t\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.433551 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.433507    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.433743 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.433596    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jss7t\" (UniqueName: \"kubernetes.io/projected/e1859d52-32a3-4a1f-a306-87f662db3581-kube-api-access-jss7t\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.433743 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.433631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-metrics-certs\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.433743 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.433656    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d52-32a3-4a1f-a306-87f662db3581-openshift-service-ca\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.433743 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:45:07.433676    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-serving-cert: secret "maas-api-e2e-ait-7083b0df-serving-cert" not found
Sep 10 15:45:07.433985 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:45:07.433791    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls podName:e1859d52-32a3-4a1f-a306-87f662db3581 nodeName:}" failed. No retries permitted until 2026-09-10 15:45:07.933767603 +0000 UTC m=+751.547894503 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls") pod "maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" (UID: "e1859d52-32a3-4a1f-a306-87f662db3581") : secret "maas-api-e2e-ait-7083b0df-serving-cert" not found
Sep 10 15:45:07.434395 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.434377    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d52-32a3-4a1f-a306-87f662db3581-openshift-service-ca\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.436272 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.436238    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-metrics-certs\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.444103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.444079    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jss7t\" (UniqueName: \"kubernetes.io/projected/e1859d52-32a3-4a1f-a306-87f662db3581-kube-api-access-jss7t\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.938429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.938396    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:07.938615 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:45:07.938585    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-serving-cert: secret "maas-api-e2e-ait-7083b0df-serving-cert" not found
Sep 10 15:45:07.938680 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:45:07.938666    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls podName:e1859d52-32a3-4a1f-a306-87f662db3581 nodeName:}" failed. No retries permitted until 2026-09-10 15:45:08.938646259 +0000 UTC m=+752.552773142 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls") pod "maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" (UID: "e1859d52-32a3-4a1f-a306-87f662db3581") : secret "maas-api-e2e-ait-7083b0df-serving-cert" not found
Sep 10 15:45:07.954144 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.954113    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"]
Sep 10 15:45:07.957796 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.957779    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:07.960195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.960169    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7083b0df-dockercfg-b7whs\""
Sep 10 15:45:07.960303 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.960226    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7083b0df\""
Sep 10 15:45:07.968982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:07.968962    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"]
Sep 10 15:45:08.039697 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.039662    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bdf9993-3ca5-4d47-9484-aff4799323f5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7083b0df-94bd57899-tq72p\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") " pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:08.039875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.039735    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9w47\" (UniqueName: \"kubernetes.io/projected/0bdf9993-3ca5-4d47-9484-aff4799323f5-kube-api-access-x9w47\") pod \"payload-processing-e2e-ait-7083b0df-94bd57899-tq72p\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") " pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:08.141261 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.141213    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bdf9993-3ca5-4d47-9484-aff4799323f5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7083b0df-94bd57899-tq72p\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") " pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:08.141415 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.141303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9w47\" (UniqueName: \"kubernetes.io/projected/0bdf9993-3ca5-4d47-9484-aff4799323f5-kube-api-access-x9w47\") pod \"payload-processing-e2e-ait-7083b0df-94bd57899-tq72p\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") " pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:08.141909 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.141888    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bdf9993-3ca5-4d47-9484-aff4799323f5-plugins-config-volume\") pod \"payload-processing-e2e-ait-7083b0df-94bd57899-tq72p\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") " pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:08.151082 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.151055    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9w47\" (UniqueName: \"kubernetes.io/projected/0bdf9993-3ca5-4d47-9484-aff4799323f5-kube-api-access-x9w47\") pod \"payload-processing-e2e-ait-7083b0df-94bd57899-tq72p\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") " pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:08.270356 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.270271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:08.542685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.542659    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"]
Sep 10 15:45:08.544381 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.544356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:45:08.544838 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:08.544817    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bdf9993_3ca5_4d47_9484_aff4799323f5.slice/crio-fec4f02176db671a51aa245ede8f0d6b72396b0503575608d79a3fceb928a9f9 WatchSource:0}: Error finding container fec4f02176db671a51aa245ede8f0d6b72396b0503575608d79a3fceb928a9f9: Status 404 returned error can't find the container with id fec4f02176db671a51aa245ede8f0d6b72396b0503575608d79a3fceb928a9f9
Sep 10 15:45:08.571437 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.571408    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p" event={"ID":"0bdf9993-3ca5-4d47-9484-aff4799323f5","Type":"ContainerStarted","Data":"fec4f02176db671a51aa245ede8f0d6b72396b0503575608d79a3fceb928a9f9"}
Sep 10 15:45:08.948677 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.948643    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:08.951143 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:08.951112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls\") pod \"maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:09.029941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:09.029908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:09.304744 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:09.304721    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"]
Sep 10 15:45:09.306292 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:09.306258    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1859d52_32a3_4a1f_a306_87f662db3581.slice/crio-bc4e7725eb883e4166d233ce943210d0f211acd0b3cc8494365d0d0d7908fd5a WatchSource:0}: Error finding container bc4e7725eb883e4166d233ce943210d0f211acd0b3cc8494365d0d0d7908fd5a: Status 404 returned error can't find the container with id bc4e7725eb883e4166d233ce943210d0f211acd0b3cc8494365d0d0d7908fd5a
Sep 10 15:45:09.579311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:09.579282    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p" event={"ID":"0bdf9993-3ca5-4d47-9484-aff4799323f5","Type":"ContainerStarted","Data":"d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685"}
Sep 10 15:45:09.579508 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:09.579398    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:09.580472 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:09.580451    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" event={"ID":"e1859d52-32a3-4a1f-a306-87f662db3581","Type":"ContainerStarted","Data":"bc4e7725eb883e4166d233ce943210d0f211acd0b3cc8494365d0d0d7908fd5a"}
Sep 10 15:45:09.603772 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:09.601056    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p" podStartSLOduration=2.601037492 podStartE2EDuration="2.601037492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:09.596221884 +0000 UTC m=+753.210348789" watchObservedRunningTime="2026-09-10 15:45:09.601037492 +0000 UTC m=+753.215164398"
Sep 10 15:45:10.587981 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:10.587948    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" event={"ID":"e1859d52-32a3-4a1f-a306-87f662db3581","Type":"ContainerStarted","Data":"f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da"}
Sep 10 15:45:10.606601 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:10.606552    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" podStartSLOduration=3.053026927 podStartE2EDuration="3.606535733s" totalImagesPullingTime="553.508806ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:10.605211508 +0000 UTC m=+754.219338417" watchObservedRunningTime="2026-09-10 15:45:10.606535733 +0000 UTC m=+754.220662636"
Sep 10 15:45:11.594102 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:11.594074    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:12.533862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:12.533835    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:45:12.534061 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:12.533918    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:45:16.563398 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.563319    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"]
Sep 10 15:45:16.566942 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.566925    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.569323 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.569303    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-d5048f-dockercfg-s2x4z\""
Sep 10 15:45:16.569423 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.569349    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-d5048f\""
Sep 10 15:45:16.577085 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.577063    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"]
Sep 10 15:45:16.602671 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.602644    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"]
Sep 10 15:45:16.606707 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.606685    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:16.616900 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.616876    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"]
Sep 10 15:45:16.713172 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.713137    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.713338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.713192    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11c9993-90d3-421f-b20b-564ebcf1784f-plugins-config-volume\") pod \"payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") " pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:16.713338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.713226    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zkbk\" (UniqueName: \"kubernetes.io/projected/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-kube-api-access-5zkbk\") pod \"payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.713338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.713250    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjd2j\" (UniqueName: \"kubernetes.io/projected/a11c9993-90d3-421f-b20b-564ebcf1784f-kube-api-access-rjd2j\") pod \"payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") " pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:16.814182 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.814096    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.814182 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.814154    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11c9993-90d3-421f-b20b-564ebcf1784f-plugins-config-volume\") pod \"payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") " pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:16.814372 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.814271    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zkbk\" (UniqueName: \"kubernetes.io/projected/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-kube-api-access-5zkbk\") pod \"payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.814372 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.814305    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjd2j\" (UniqueName: \"kubernetes.io/projected/a11c9993-90d3-421f-b20b-564ebcf1784f-kube-api-access-rjd2j\") pod \"payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") " pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:16.814878 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.814860    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.814933 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.814865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11c9993-90d3-421f-b20b-564ebcf1784f-plugins-config-volume\") pod \"payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") " pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:16.822213 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.822189    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zkbk\" (UniqueName: \"kubernetes.io/projected/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-kube-api-access-5zkbk\") pod \"payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.822321 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.822222    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjd2j\" (UniqueName: \"kubernetes.io/projected/a11c9993-90d3-421f-b20b-564ebcf1784f-kube-api-access-rjd2j\") pod \"payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") " pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:16.878263 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.878229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:16.918959 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:16.918931    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:17.157078 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.157055    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"]
Sep 10 15:45:17.158171 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:17.158144    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d11f33_2408_48e8_a2fd_05bbb98ba3f9.slice/crio-b062e8e04faa1f792f1aec1d12462c020b1d707d8f6abc9ec99c7cd75207d051 WatchSource:0}: Error finding container b062e8e04faa1f792f1aec1d12462c020b1d707d8f6abc9ec99c7cd75207d051: Status 404 returned error can't find the container with id b062e8e04faa1f792f1aec1d12462c020b1d707d8f6abc9ec99c7cd75207d051
Sep 10 15:45:17.210454 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.210422    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"]
Sep 10 15:45:17.211774 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:17.211729    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda11c9993_90d3_421f_b20b_564ebcf1784f.slice/crio-e8b29a2636195db2088b1cdfd2a01df3b8a4a0c1a9c7edd45ee695f13dc228a1 WatchSource:0}: Error finding container e8b29a2636195db2088b1cdfd2a01df3b8a4a0c1a9c7edd45ee695f13dc228a1: Status 404 returned error can't find the container with id e8b29a2636195db2088b1cdfd2a01df3b8a4a0c1a9c7edd45ee695f13dc228a1
Sep 10 15:45:17.606058 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.606007    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:17.632854 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.632818    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx" event={"ID":"06d11f33-2408-48e8-a2fd-05bbb98ba3f9","Type":"ContainerStarted","Data":"f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4"}
Sep 10 15:45:17.633032 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.632872    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx" event={"ID":"06d11f33-2408-48e8-a2fd-05bbb98ba3f9","Type":"ContainerStarted","Data":"b062e8e04faa1f792f1aec1d12462c020b1d707d8f6abc9ec99c7cd75207d051"}
Sep 10 15:45:17.633032 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.632917    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:17.635265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.635237    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz" event={"ID":"a11c9993-90d3-421f-b20b-564ebcf1784f","Type":"ContainerStarted","Data":"e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed"}
Sep 10 15:45:17.635407 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.635294    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz" event={"ID":"a11c9993-90d3-421f-b20b-564ebcf1784f","Type":"ContainerStarted","Data":"e8b29a2636195db2088b1cdfd2a01df3b8a4a0c1a9c7edd45ee695f13dc228a1"}
Sep 10 15:45:17.635761 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.635718    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:17.651526 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.651483    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx" podStartSLOduration=1.6514664209999999 podStartE2EDuration="1.651466421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:17.650027553 +0000 UTC m=+761.264154459" watchObservedRunningTime="2026-09-10 15:45:17.651466421 +0000 UTC m=+761.265593378"
Sep 10 15:45:17.666507 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:17.666461    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz" podStartSLOduration=1.6664430590000001 podStartE2EDuration="1.666443059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:17.664609697 +0000 UTC m=+761.278736603" watchObservedRunningTime="2026-09-10 15:45:17.666443059 +0000 UTC m=+761.280569963"
Sep 10 15:45:20.591138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:20.591103    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:28.643202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:28.643170    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:45:28.643594 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:28.643296    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:45:29.760499 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:29.760470    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"]
Sep 10 15:45:29.765021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:29.765003    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:29.767502 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:29.767475    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a850d-dockercfg-8mp4h\""
Sep 10 15:45:29.767684 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:29.767672    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a850d\""
Sep 10 15:45:29.773265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:29.773239    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"]
Sep 10 15:45:29.936625 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:29.936586    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81cc378d-f314-45d7-aced-e3e742d75c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:29.936842 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:29.936723    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47fjb\" (UniqueName: \"kubernetes.io/projected/81cc378d-f314-45d7-aced-e3e742d75c39-kube-api-access-47fjb\") pod \"payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:30.038076 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.037993    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47fjb\" (UniqueName: \"kubernetes.io/projected/81cc378d-f314-45d7-aced-e3e742d75c39-kube-api-access-47fjb\") pod \"payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:30.038076 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.038044    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81cc378d-f314-45d7-aced-e3e742d75c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:30.038644 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.038626    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81cc378d-f314-45d7-aced-e3e742d75c39-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:30.045908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.045887    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47fjb\" (UniqueName: \"kubernetes.io/projected/81cc378d-f314-45d7-aced-e3e742d75c39-kube-api-access-47fjb\") pod \"payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:30.079356 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.079327    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:30.352571 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.352547    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"]
Sep 10 15:45:30.354602 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:30.354577    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81cc378d_f314_45d7_aced_e3e742d75c39.slice/crio-eb8cf80bb8170d29444f278f8386d663273658ff59f6cb36c50945d4b8b99e4d WatchSource:0}: Error finding container eb8cf80bb8170d29444f278f8386d663273658ff59f6cb36c50945d4b8b99e4d: Status 404 returned error can't find the container with id eb8cf80bb8170d29444f278f8386d663273658ff59f6cb36c50945d4b8b99e4d
Sep 10 15:45:30.717799 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.717701    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p" event={"ID":"81cc378d-f314-45d7-aced-e3e742d75c39","Type":"ContainerStarted","Data":"befdc7341542dcf83b653ca22416b9105cc4111c4c02e3d03c831fbe5eb01d81"}
Sep 10 15:45:30.717799 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.717778    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p" event={"ID":"81cc378d-f314-45d7-aced-e3e742d75c39","Type":"ContainerStarted","Data":"eb8cf80bb8170d29444f278f8386d663273658ff59f6cb36c50945d4b8b99e4d"}
Sep 10 15:45:30.718058 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.717810    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:30.734394 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:30.734349    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p" podStartSLOduration=1.7343282850000001 podStartE2EDuration="1.734328285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:30.733080593 +0000 UTC m=+774.347207521" watchObservedRunningTime="2026-09-10 15:45:30.734328285 +0000 UTC m=+774.348455244"
Sep 10 15:45:41.726169 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:41.726135    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:45:48.238857 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.238820    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"]
Sep 10 15:45:48.239324 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.239050    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" podUID="e1859d52-32a3-4a1f-a306-87f662db3581" containerName="maas-api" containerID="cri-o://f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da" gracePeriod=30
Sep 10 15:45:48.299458 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.299414    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"]
Sep 10 15:45:48.299703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.299675    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p" podUID="0bdf9993-3ca5-4d47-9484-aff4799323f5" containerName="payload-processing" containerID="cri-o://d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685" gracePeriod=30
Sep 10 15:45:48.543988 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.543962    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:48.574970 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.574948    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:48.591101 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.589792    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls\") pod \"e1859d52-32a3-4a1f-a306-87f662db3581\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") "
Sep 10 15:45:48.591101 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.590176    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-metrics-certs\") pod \"e1859d52-32a3-4a1f-a306-87f662db3581\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") "
Sep 10 15:45:48.591101 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.590213    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jss7t\" (UniqueName: \"kubernetes.io/projected/e1859d52-32a3-4a1f-a306-87f662db3581-kube-api-access-jss7t\") pod \"e1859d52-32a3-4a1f-a306-87f662db3581\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") "
Sep 10 15:45:48.591101 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.590291    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d52-32a3-4a1f-a306-87f662db3581-openshift-service-ca\") pod \"e1859d52-32a3-4a1f-a306-87f662db3581\" (UID: \"e1859d52-32a3-4a1f-a306-87f662db3581\") "
Sep 10 15:45:48.591101 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.590895    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1859d52-32a3-4a1f-a306-87f662db3581-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e1859d52-32a3-4a1f-a306-87f662db3581" (UID: "e1859d52-32a3-4a1f-a306-87f662db3581"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:48.596983 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.596950    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e1859d52-32a3-4a1f-a306-87f662db3581" (UID: "e1859d52-32a3-4a1f-a306-87f662db3581"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:48.597879 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.597850    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1859d52-32a3-4a1f-a306-87f662db3581-kube-api-access-jss7t" (OuterVolumeSpecName: "kube-api-access-jss7t") pod "e1859d52-32a3-4a1f-a306-87f662db3581" (UID: "e1859d52-32a3-4a1f-a306-87f662db3581"). InnerVolumeSpecName "kube-api-access-jss7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:48.599040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.599008    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e1859d52-32a3-4a1f-a306-87f662db3581" (UID: "e1859d52-32a3-4a1f-a306-87f662db3581"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:45:48.691106 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.691027    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x9w47\" (UniqueName: \"kubernetes.io/projected/0bdf9993-3ca5-4d47-9484-aff4799323f5-kube-api-access-x9w47\") pod \"0bdf9993-3ca5-4d47-9484-aff4799323f5\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") "
Sep 10 15:45:48.691307 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.691154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bdf9993-3ca5-4d47-9484-aff4799323f5-plugins-config-volume\") pod \"0bdf9993-3ca5-4d47-9484-aff4799323f5\" (UID: \"0bdf9993-3ca5-4d47-9484-aff4799323f5\") "
Sep 10 15:45:48.691538 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.691514    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bdf9993-3ca5-4d47-9484-aff4799323f5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0bdf9993-3ca5-4d47-9484-aff4799323f5" (UID: "0bdf9993-3ca5-4d47-9484-aff4799323f5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:45:48.691690 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.691672    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:45:48.691784 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.691695    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e1859d52-32a3-4a1f-a306-87f662db3581-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:45:48.692095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.691711    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jss7t\" (UniqueName: \"kubernetes.io/projected/e1859d52-32a3-4a1f-a306-87f662db3581-kube-api-access-jss7t\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:45:48.692181 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.692106    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0bdf9993-3ca5-4d47-9484-aff4799323f5-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:45:48.692181 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.692128    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e1859d52-32a3-4a1f-a306-87f662db3581-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:45:48.693549 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.693527    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bdf9993-3ca5-4d47-9484-aff4799323f5-kube-api-access-x9w47" (OuterVolumeSpecName: "kube-api-access-x9w47") pod "0bdf9993-3ca5-4d47-9484-aff4799323f5" (UID: "0bdf9993-3ca5-4d47-9484-aff4799323f5"). InnerVolumeSpecName "kube-api-access-x9w47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:45:48.793022 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.792933    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9w47\" (UniqueName: \"kubernetes.io/projected/0bdf9993-3ca5-4d47-9484-aff4799323f5-kube-api-access-x9w47\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:45:48.823134 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.823104    2577 generic.go:356] "Generic (PLEG): container finished" podID="0bdf9993-3ca5-4d47-9484-aff4799323f5" containerID="d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685" exitCode=0
Sep 10 15:45:48.823305 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.823150    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"
Sep 10 15:45:48.823305 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.823173    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p" event={"ID":"0bdf9993-3ca5-4d47-9484-aff4799323f5","Type":"ContainerDied","Data":"d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685"}
Sep 10 15:45:48.823305 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.823225    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p" event={"ID":"0bdf9993-3ca5-4d47-9484-aff4799323f5","Type":"ContainerDied","Data":"fec4f02176db671a51aa245ede8f0d6b72396b0503575608d79a3fceb928a9f9"}
Sep 10 15:45:48.823305 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.823247    2577 scope.go:117] "RemoveContainer" containerID="d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685"
Sep 10 15:45:48.824577 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.824553    2577 generic.go:356] "Generic (PLEG): container finished" podID="e1859d52-32a3-4a1f-a306-87f662db3581" containerID="f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da" exitCode=0
Sep 10 15:45:48.824719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.824620    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"
Sep 10 15:45:48.824719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.824642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" event={"ID":"e1859d52-32a3-4a1f-a306-87f662db3581","Type":"ContainerDied","Data":"f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da"}
Sep 10 15:45:48.824719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.824682    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h" event={"ID":"e1859d52-32a3-4a1f-a306-87f662db3581","Type":"ContainerDied","Data":"bc4e7725eb883e4166d233ce943210d0f211acd0b3cc8494365d0d0d7908fd5a"}
Sep 10 15:45:48.833990 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.833975    2577 scope.go:117] "RemoveContainer" containerID="d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685"
Sep 10 15:45:48.834297 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:45:48.834273    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685\": container with ID starting with d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685 not found: ID does not exist" containerID="d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685"
Sep 10 15:45:48.834386 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.834309    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685"} err="failed to get container status \"d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685\": rpc error: code = NotFound desc = could not find container \"d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685\": container with ID starting with d8eb009a5ab79770d1b062ff50c6229890a19e7f4a9754c21142ff3cf0629685 not found: ID does not exist"
Sep 10 15:45:48.834386 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.834332    2577 scope.go:117] "RemoveContainer" containerID="f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da"
Sep 10 15:45:48.842480 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.842464    2577 scope.go:117] "RemoveContainer" containerID="f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da"
Sep 10 15:45:48.842722 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:45:48.842703    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da\": container with ID starting with f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da not found: ID does not exist" containerID="f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da"
Sep 10 15:45:48.842793 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.842727    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da"} err="failed to get container status \"f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da\": rpc error: code = NotFound desc = could not find container \"f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da\": container with ID starting with f76c8fa822c2e1f3c6fa8179772e9bc40430a93694091b8cb92e8bb0fc9180da not found: ID does not exist"
Sep 10 15:45:48.846191 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.846173    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"]
Sep 10 15:45:48.848388 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.848368    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7083b0df-94bd57899-tq72p"]
Sep 10 15:45:48.858189 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.858166    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"]
Sep 10 15:45:48.861832 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.861807    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-7083b0df-77c4d5b4d5-tqg9h"]
Sep 10 15:45:48.907986 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.907958    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0bdf9993-3ca5-4d47-9484-aff4799323f5" path="/var/lib/kubelet/pods/0bdf9993-3ca5-4d47-9484-aff4799323f5/volumes"
Sep 10 15:45:48.908351 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:48.908339    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1859d52-32a3-4a1f-a306-87f662db3581" path="/var/lib/kubelet/pods/e1859d52-32a3-4a1f-a306-87f662db3581/volumes"
Sep 10 15:45:55.581927 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.581889    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"]
Sep 10 15:45:55.582373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.582276    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0bdf9993-3ca5-4d47-9484-aff4799323f5" containerName="payload-processing"
Sep 10 15:45:55.582373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.582287    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0bdf9993-3ca5-4d47-9484-aff4799323f5" containerName="payload-processing"
Sep 10 15:45:55.582373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.582300    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1859d52-32a3-4a1f-a306-87f662db3581" containerName="maas-api"
Sep 10 15:45:55.582373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.582305    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1859d52-32a3-4a1f-a306-87f662db3581" containerName="maas-api"
Sep 10 15:45:55.582578 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.582379    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0bdf9993-3ca5-4d47-9484-aff4799323f5" containerName="payload-processing"
Sep 10 15:45:55.582578 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.582387    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1859d52-32a3-4a1f-a306-87f662db3581" containerName="maas-api"
Sep 10 15:45:55.585813 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.585791    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.592672 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.592650    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-d8392afe-serving-cert\""
Sep 10 15:45:55.602260 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.602199    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"]
Sep 10 15:45:55.652691 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.652658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e373289e-f642-40d6-8864-119ced8eb3a9-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.652860 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.652697    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-metrics-certs\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.652860 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.652843    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-maas-api-tls\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.652938 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.652890    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798jp\" (UniqueName: \"kubernetes.io/projected/e373289e-f642-40d6-8864-119ced8eb3a9-kube-api-access-798jp\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.753543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.753478    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-maas-api-tls\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.753543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.753552    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798jp\" (UniqueName: \"kubernetes.io/projected/e373289e-f642-40d6-8864-119ced8eb3a9-kube-api-access-798jp\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.753862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.753586    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e373289e-f642-40d6-8864-119ced8eb3a9-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.753862 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.753609    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-metrics-certs\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.754373 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.754346    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e373289e-f642-40d6-8864-119ced8eb3a9-openshift-service-ca\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.756146 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.756119    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-maas-api-tls\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.756235 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.756154    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-metrics-certs\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.764080 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.764060    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-798jp\" (UniqueName: \"kubernetes.io/projected/e373289e-f642-40d6-8864-119ced8eb3a9-kube-api-access-798jp\") pod \"maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:55.918633 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:55.918546    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:56.198392 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.198353    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"]
Sep 10 15:45:56.199582 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:56.199551    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode373289e_f642_40d6_8864_119ced8eb3a9.slice/crio-2e8da621164f81f71d471ac073fc1856568869db7e848bd9b736103460b67813 WatchSource:0}: Error finding container 2e8da621164f81f71d471ac073fc1856568869db7e848bd9b736103460b67813: Status 404 returned error can't find the container with id 2e8da621164f81f71d471ac073fc1856568869db7e848bd9b736103460b67813
Sep 10 15:45:56.220416 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.220391    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"]
Sep 10 15:45:56.225142 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.225127    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.227458 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.227442    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-d8392afe\""
Sep 10 15:45:56.227458 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.227449    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-d8392afe-dockercfg-ppkvh\""
Sep 10 15:45:56.232013 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.231992    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"]
Sep 10 15:45:56.258976 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.258943    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba9841-3279-4710-a2b0-e234e6ee68c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.259163 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.259055    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6bdv\" (UniqueName: \"kubernetes.io/projected/d5ba9841-3279-4710-a2b0-e234e6ee68c9-kube-api-access-b6bdv\") pod \"payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.359969 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.359931    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6bdv\" (UniqueName: \"kubernetes.io/projected/d5ba9841-3279-4710-a2b0-e234e6ee68c9-kube-api-access-b6bdv\") pod \"payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.360139 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.360022    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba9841-3279-4710-a2b0-e234e6ee68c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.360706 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.360687    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba9841-3279-4710-a2b0-e234e6ee68c9-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.368432 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.368401    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6bdv\" (UniqueName: \"kubernetes.io/projected/d5ba9841-3279-4710-a2b0-e234e6ee68c9-kube-api-access-b6bdv\") pod \"payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.537513 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.537432    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:56.807921 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.807895    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"]
Sep 10 15:45:56.809707 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:45:56.809683    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ba9841_3279_4710_a2b0_e234e6ee68c9.slice/crio-c9a90b6b85847a2d78f07d3e05b57c57c6c387e3942e8c928d2b4d8856bdb56c WatchSource:0}: Error finding container c9a90b6b85847a2d78f07d3e05b57c57c6c387e3942e8c928d2b4d8856bdb56c: Status 404 returned error can't find the container with id c9a90b6b85847a2d78f07d3e05b57c57c6c387e3942e8c928d2b4d8856bdb56c
Sep 10 15:45:56.872042 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.872014    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk" event={"ID":"e373289e-f642-40d6-8864-119ced8eb3a9","Type":"ContainerStarted","Data":"2e8da621164f81f71d471ac073fc1856568869db7e848bd9b736103460b67813"}
Sep 10 15:45:56.873717 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:56.873694    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc" event={"ID":"d5ba9841-3279-4710-a2b0-e234e6ee68c9","Type":"ContainerStarted","Data":"c9a90b6b85847a2d78f07d3e05b57c57c6c387e3942e8c928d2b4d8856bdb56c"}
Sep 10 15:45:57.881001 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:57.880917    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc" event={"ID":"d5ba9841-3279-4710-a2b0-e234e6ee68c9","Type":"ContainerStarted","Data":"92943388071f4d5bd0ad8294a339e9ff91851e3030d3a415c20fab9afd971a0c"}
Sep 10 15:45:57.881457 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:57.881053    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:45:57.898085 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:57.898030    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc" podStartSLOduration=1.898011004 podStartE2EDuration="1.898011004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:57.895722467 +0000 UTC m=+801.509849372" watchObservedRunningTime="2026-09-10 15:45:57.898011004 +0000 UTC m=+801.512137907"
Sep 10 15:45:58.887665 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:58.887628    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk" event={"ID":"e373289e-f642-40d6-8864-119ced8eb3a9","Type":"ContainerStarted","Data":"91a854674f308d7e4469244fdd8e55398bbb080f8f39b1b9d6afd33baf3d4528"}
Sep 10 15:45:58.888180 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:58.887938    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:45:58.911127 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:45:58.911067    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk" podStartSLOduration=1.9699731169999999 podStartE2EDuration="3.911051841s" totalImagesPullingTime="1.941078724s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:45:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:45:58.9088517 +0000 UTC m=+802.522978605" watchObservedRunningTime="2026-09-10 15:45:58.911051841 +0000 UTC m=+802.525178745"
Sep 10 15:46:04.898943 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:04.898913    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:46:08.889818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:08.889782    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:46:26.839379 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:26.839345    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"]
Sep 10 15:46:26.839873 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:26.839617    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk" podUID="e373289e-f642-40d6-8864-119ced8eb3a9" containerName="maas-api" containerID="cri-o://91a854674f308d7e4469244fdd8e55398bbb080f8f39b1b9d6afd33baf3d4528" gracePeriod=30
Sep 10 15:46:26.911154 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:26.911125    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"]
Sep 10 15:46:26.911374 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:26.911332    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc" podUID="d5ba9841-3279-4710-a2b0-e234e6ee68c9" containerName="payload-pre-processing" containerID="cri-o://92943388071f4d5bd0ad8294a339e9ff91851e3030d3a415c20fab9afd971a0c" gracePeriod=30
Sep 10 15:46:27.050484 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.049690    2577 generic.go:356] "Generic (PLEG): container finished" podID="e373289e-f642-40d6-8864-119ced8eb3a9" containerID="91a854674f308d7e4469244fdd8e55398bbb080f8f39b1b9d6afd33baf3d4528" exitCode=0
Sep 10 15:46:27.050484 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.049807    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk" event={"ID":"e373289e-f642-40d6-8864-119ced8eb3a9","Type":"ContainerDied","Data":"91a854674f308d7e4469244fdd8e55398bbb080f8f39b1b9d6afd33baf3d4528"}
Sep 10 15:46:27.052617 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.052591    2577 generic.go:356] "Generic (PLEG): container finished" podID="d5ba9841-3279-4710-a2b0-e234e6ee68c9" containerID="92943388071f4d5bd0ad8294a339e9ff91851e3030d3a415c20fab9afd971a0c" exitCode=0
Sep 10 15:46:27.052786 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.052667    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc" event={"ID":"d5ba9841-3279-4710-a2b0-e234e6ee68c9","Type":"ContainerDied","Data":"92943388071f4d5bd0ad8294a339e9ff91851e3030d3a415c20fab9afd971a0c"}
Sep 10 15:46:27.166591 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.166568    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:46:27.211828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.211805    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:46:27.242424 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.242392    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba9841-3279-4710-a2b0-e234e6ee68c9-plugins-config-volume\") pod \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") "
Sep 10 15:46:27.242603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.242435    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-maas-api-tls\") pod \"e373289e-f642-40d6-8864-119ced8eb3a9\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") "
Sep 10 15:46:27.242603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.242462    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b6bdv\" (UniqueName: \"kubernetes.io/projected/d5ba9841-3279-4710-a2b0-e234e6ee68c9-kube-api-access-b6bdv\") pod \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\" (UID: \"d5ba9841-3279-4710-a2b0-e234e6ee68c9\") "
Sep 10 15:46:27.242603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.242513    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-metrics-certs\") pod \"e373289e-f642-40d6-8864-119ced8eb3a9\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") "
Sep 10 15:46:27.242603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.242574    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e373289e-f642-40d6-8864-119ced8eb3a9-openshift-service-ca\") pod \"e373289e-f642-40d6-8864-119ced8eb3a9\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") "
Sep 10 15:46:27.242879 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.242838    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ba9841-3279-4710-a2b0-e234e6ee68c9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5ba9841-3279-4710-a2b0-e234e6ee68c9" (UID: "d5ba9841-3279-4710-a2b0-e234e6ee68c9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:27.243009 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.242987    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-798jp\" (UniqueName: \"kubernetes.io/projected/e373289e-f642-40d6-8864-119ced8eb3a9-kube-api-access-798jp\") pod \"e373289e-f642-40d6-8864-119ced8eb3a9\" (UID: \"e373289e-f642-40d6-8864-119ced8eb3a9\") "
Sep 10 15:46:27.243229 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.243201    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e373289e-f642-40d6-8864-119ced8eb3a9-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e373289e-f642-40d6-8864-119ced8eb3a9" (UID: "e373289e-f642-40d6-8864-119ced8eb3a9"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:46:27.243516 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.243498    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e373289e-f642-40d6-8864-119ced8eb3a9-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:46:27.243588 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.243522    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ba9841-3279-4710-a2b0-e234e6ee68c9-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:46:27.245020 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.244992    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ba9841-3279-4710-a2b0-e234e6ee68c9-kube-api-access-b6bdv" (OuterVolumeSpecName: "kube-api-access-b6bdv") pod "d5ba9841-3279-4710-a2b0-e234e6ee68c9" (UID: "d5ba9841-3279-4710-a2b0-e234e6ee68c9"). InnerVolumeSpecName "kube-api-access-b6bdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:27.245020 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.244994    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e373289e-f642-40d6-8864-119ced8eb3a9" (UID: "e373289e-f642-40d6-8864-119ced8eb3a9"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:27.245275 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.245257    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e373289e-f642-40d6-8864-119ced8eb3a9" (UID: "e373289e-f642-40d6-8864-119ced8eb3a9"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:46:27.245627 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.245613    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e373289e-f642-40d6-8864-119ced8eb3a9-kube-api-access-798jp" (OuterVolumeSpecName: "kube-api-access-798jp") pod "e373289e-f642-40d6-8864-119ced8eb3a9" (UID: "e373289e-f642-40d6-8864-119ced8eb3a9"). InnerVolumeSpecName "kube-api-access-798jp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:46:27.344541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.344500    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-798jp\" (UniqueName: \"kubernetes.io/projected/e373289e-f642-40d6-8864-119ced8eb3a9-kube-api-access-798jp\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:46:27.344541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.344533    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:46:27.344541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.344543    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b6bdv\" (UniqueName: \"kubernetes.io/projected/d5ba9841-3279-4710-a2b0-e234e6ee68c9-kube-api-access-b6bdv\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:46:27.344541 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:27.344552    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e373289e-f642-40d6-8864-119ced8eb3a9-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:46:28.060923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.060888    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk" event={"ID":"e373289e-f642-40d6-8864-119ced8eb3a9","Type":"ContainerDied","Data":"2e8da621164f81f71d471ac073fc1856568869db7e848bd9b736103460b67813"}
Sep 10 15:46:28.060923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.060911    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"
Sep 10 15:46:28.061440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.060965    2577 scope.go:117] "RemoveContainer" containerID="91a854674f308d7e4469244fdd8e55398bbb080f8f39b1b9d6afd33baf3d4528"
Sep 10 15:46:28.062680 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.062649    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc" event={"ID":"d5ba9841-3279-4710-a2b0-e234e6ee68c9","Type":"ContainerDied","Data":"c9a90b6b85847a2d78f07d3e05b57c57c6c387e3942e8c928d2b4d8856bdb56c"}
Sep 10 15:46:28.062937 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.062914    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"
Sep 10 15:46:28.074300 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.074077    2577 scope.go:117] "RemoveContainer" containerID="92943388071f4d5bd0ad8294a339e9ff91851e3030d3a415c20fab9afd971a0c"
Sep 10 15:46:28.085699 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.085678    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"]
Sep 10 15:46:28.094010 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.093981    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-d8392afe-76dcc48744-s6hvk"]
Sep 10 15:46:28.104941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.104890    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"]
Sep 10 15:46:28.110694 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.110673    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-d8392afe-56d79847d6-ft8fc"]
Sep 10 15:46:28.908264 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.908225    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ba9841-3279-4710-a2b0-e234e6ee68c9" path="/var/lib/kubelet/pods/d5ba9841-3279-4710-a2b0-e234e6ee68c9/volumes"
Sep 10 15:46:28.908781 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:28.908741    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e373289e-f642-40d6-8864-119ced8eb3a9" path="/var/lib/kubelet/pods/e373289e-f642-40d6-8864-119ced8eb3a9/volumes"
Sep 10 15:46:31.729767 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.729730    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"]
Sep 10 15:46:31.730265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.730192    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e373289e-f642-40d6-8864-119ced8eb3a9" containerName="maas-api"
Sep 10 15:46:31.730265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.730211    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="e373289e-f642-40d6-8864-119ced8eb3a9" containerName="maas-api"
Sep 10 15:46:31.730265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.730223    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ba9841-3279-4710-a2b0-e234e6ee68c9" containerName="payload-pre-processing"
Sep 10 15:46:31.730265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.730229    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ba9841-3279-4710-a2b0-e234e6ee68c9" containerName="payload-pre-processing"
Sep 10 15:46:31.730470 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.730300    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="e373289e-f642-40d6-8864-119ced8eb3a9" containerName="maas-api"
Sep 10 15:46:31.730470 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.730308    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ba9841-3279-4710-a2b0-e234e6ee68c9" containerName="payload-pre-processing"
Sep 10 15:46:31.733464 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.733437    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.735780 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.735741    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-96d7ee49-serving-cert\""
Sep 10 15:46:31.737859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.737839    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"]
Sep 10 15:46:31.782102 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.782072    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdzzh\" (UniqueName: \"kubernetes.io/projected/0881ccfc-caa1-4d40-a0a2-b777d2c89654-kube-api-access-wdzzh\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.782276 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.782107    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-metrics-certs\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.782276 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.782158    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881ccfc-caa1-4d40-a0a2-b777d2c89654-openshift-service-ca\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.782276 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.782211    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-maas-api-tls\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.883646 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.883614    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881ccfc-caa1-4d40-a0a2-b777d2c89654-openshift-service-ca\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.883646 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.883668    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-maas-api-tls\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.883646 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.883707    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdzzh\" (UniqueName: \"kubernetes.io/projected/0881ccfc-caa1-4d40-a0a2-b777d2c89654-kube-api-access-wdzzh\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.883977 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.883732    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-metrics-certs\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.884291 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.884265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881ccfc-caa1-4d40-a0a2-b777d2c89654-openshift-service-ca\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.886284 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.886256    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-maas-api-tls\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.886451 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.886429    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-metrics-certs\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:31.892395 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:31.892371    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdzzh\" (UniqueName: \"kubernetes.io/projected/0881ccfc-caa1-4d40-a0a2-b777d2c89654-kube-api-access-wdzzh\") pod \"maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:32.047267 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.047181    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:32.323922 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.323893    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"]
Sep 10 15:46:32.325375 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:46:32.325346    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0881ccfc_caa1_4d40_a0a2_b777d2c89654.slice/crio-564fe8183c9b4330b9300845957cfc841f83eb579e3056b1d5e9394bfe329b50 WatchSource:0}: Error finding container 564fe8183c9b4330b9300845957cfc841f83eb579e3056b1d5e9394bfe329b50: Status 404 returned error can't find the container with id 564fe8183c9b4330b9300845957cfc841f83eb579e3056b1d5e9394bfe329b50
Sep 10 15:46:32.354815 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.354789    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"]
Sep 10 15:46:32.359294 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.359279    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.361695 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.361673    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-96d7ee49\""
Sep 10 15:46:32.361821 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.361792    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-96d7ee49-dockercfg-hz8v4\""
Sep 10 15:46:32.369665 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.369635    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"]
Sep 10 15:46:32.388330 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.388305    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.388457 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.388349    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rp69\" (UniqueName: \"kubernetes.io/projected/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-kube-api-access-9rp69\") pod \"payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.489870 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.489828    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.490033 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.489906    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rp69\" (UniqueName: \"kubernetes.io/projected/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-kube-api-access-9rp69\") pod \"payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.490792 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.490743    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.500141 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.500106    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rp69\" (UniqueName: \"kubernetes.io/projected/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-kube-api-access-9rp69\") pod \"payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.670803 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.670744    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:32.948172 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:32.948146    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"]
Sep 10 15:46:32.949614 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:46:32.949587    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b882b9_8e9e_4fce_a79f_23d1cc39f4bc.slice/crio-c105d9fd69ad4fe1af580fc64020bbe0001515c09f6f2118f259a56e1fe9efa0 WatchSource:0}: Error finding container c105d9fd69ad4fe1af580fc64020bbe0001515c09f6f2118f259a56e1fe9efa0: Status 404 returned error can't find the container with id c105d9fd69ad4fe1af580fc64020bbe0001515c09f6f2118f259a56e1fe9efa0
Sep 10 15:46:33.098713 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.098681    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv" event={"ID":"0881ccfc-caa1-4d40-a0a2-b777d2c89654","Type":"ContainerStarted","Data":"acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858"}
Sep 10 15:46:33.098941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.098743    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv" event={"ID":"0881ccfc-caa1-4d40-a0a2-b777d2c89654","Type":"ContainerStarted","Data":"564fe8183c9b4330b9300845957cfc841f83eb579e3056b1d5e9394bfe329b50"}
Sep 10 15:46:33.098941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.098882    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:33.101062 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.101034    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr" event={"ID":"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc","Type":"ContainerStarted","Data":"808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8"}
Sep 10 15:46:33.101234 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.101089    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr" event={"ID":"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc","Type":"ContainerStarted","Data":"c105d9fd69ad4fe1af580fc64020bbe0001515c09f6f2118f259a56e1fe9efa0"}
Sep 10 15:46:33.101234 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.101113    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:46:33.116613 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.116566    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv" podStartSLOduration=1.725212231 podStartE2EDuration="2.116552243s" totalImagesPullingTime="391.340012ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:33.113438673 +0000 UTC m=+836.727565576" watchObservedRunningTime="2026-09-10 15:46:33.116552243 +0000 UTC m=+836.730679147"
Sep 10 15:46:33.130941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:33.130890    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr" podStartSLOduration=1.130876537 podStartE2EDuration="1.130876537s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:46:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:46:33.128419043 +0000 UTC m=+836.742545948" watchObservedRunningTime="2026-09-10 15:46:33.130876537 +0000 UTC m=+836.745003440"
Sep 10 15:46:39.114480 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:39.114452    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:46:44.111574 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:46:44.111542    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:47:02.611641 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:02.611607    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"]
Sep 10 15:47:02.612135 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:02.611863    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv" podUID="0881ccfc-caa1-4d40-a0a2-b777d2c89654" containerName="maas-api" containerID="cri-o://acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858" gracePeriod=30
Sep 10 15:47:02.679816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:02.679784    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"]
Sep 10 15:47:02.680044 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:02.680018    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr" podUID="49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" containerName="payload-pre-processing" containerID="cri-o://808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8" gracePeriod=30
Sep 10 15:47:02.936545 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:02.936521    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:47:02.960719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:02.960695    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:47:03.062440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.062400    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881ccfc-caa1-4d40-a0a2-b777d2c89654-openshift-service-ca\") pod \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") "
Sep 10 15:47:03.062440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.062445    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rp69\" (UniqueName: \"kubernetes.io/projected/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-kube-api-access-9rp69\") pod \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") "
Sep 10 15:47:03.062688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.062489    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wdzzh\" (UniqueName: \"kubernetes.io/projected/0881ccfc-caa1-4d40-a0a2-b777d2c89654-kube-api-access-wdzzh\") pod \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") "
Sep 10 15:47:03.062688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.062516    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-maas-api-tls\") pod \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") "
Sep 10 15:47:03.062688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.062552    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-plugins-config-volume\") pod \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\" (UID: \"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc\") "
Sep 10 15:47:03.062688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.062586    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-metrics-certs\") pod \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\" (UID: \"0881ccfc-caa1-4d40-a0a2-b777d2c89654\") "
Sep 10 15:47:03.062950 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.062893    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0881ccfc-caa1-4d40-a0a2-b777d2c89654-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0881ccfc-caa1-4d40-a0a2-b777d2c89654" (UID: "0881ccfc-caa1-4d40-a0a2-b777d2c89654"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:47:03.063031 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.063001    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" (UID: "49b882b9-8e9e-4fce-a79f-23d1cc39f4bc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:47:03.064877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.064845    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-kube-api-access-9rp69" (OuterVolumeSpecName: "kube-api-access-9rp69") pod "49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" (UID: "49b882b9-8e9e-4fce-a79f-23d1cc39f4bc"). InnerVolumeSpecName "kube-api-access-9rp69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:03.064978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.064901    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0881ccfc-caa1-4d40-a0a2-b777d2c89654" (UID: "0881ccfc-caa1-4d40-a0a2-b777d2c89654"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:47:03.064978 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.064912    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0881ccfc-caa1-4d40-a0a2-b777d2c89654-kube-api-access-wdzzh" (OuterVolumeSpecName: "kube-api-access-wdzzh") pod "0881ccfc-caa1-4d40-a0a2-b777d2c89654" (UID: "0881ccfc-caa1-4d40-a0a2-b777d2c89654"). InnerVolumeSpecName "kube-api-access-wdzzh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:03.065128 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.065109    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0881ccfc-caa1-4d40-a0a2-b777d2c89654" (UID: "0881ccfc-caa1-4d40-a0a2-b777d2c89654"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:47:03.163843 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.163731    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:03.163843 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.163795    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:03.163843 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.163804    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0881ccfc-caa1-4d40-a0a2-b777d2c89654-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:03.163843 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.163813    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rp69\" (UniqueName: \"kubernetes.io/projected/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc-kube-api-access-9rp69\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:03.163843 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.163822    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wdzzh\" (UniqueName: \"kubernetes.io/projected/0881ccfc-caa1-4d40-a0a2-b777d2c89654-kube-api-access-wdzzh\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:03.163843 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.163830    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0881ccfc-caa1-4d40-a0a2-b777d2c89654-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:03.273987 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.273948    2577 generic.go:356] "Generic (PLEG): container finished" podID="0881ccfc-caa1-4d40-a0a2-b777d2c89654" containerID="acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858" exitCode=0
Sep 10 15:47:03.273987 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.273990    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"
Sep 10 15:47:03.274207 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.274026    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv" event={"ID":"0881ccfc-caa1-4d40-a0a2-b777d2c89654","Type":"ContainerDied","Data":"acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858"}
Sep 10 15:47:03.274207 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.274076    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv" event={"ID":"0881ccfc-caa1-4d40-a0a2-b777d2c89654","Type":"ContainerDied","Data":"564fe8183c9b4330b9300845957cfc841f83eb579e3056b1d5e9394bfe329b50"}
Sep 10 15:47:03.274207 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.274098    2577 scope.go:117] "RemoveContainer" containerID="acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858"
Sep 10 15:47:03.275429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.275408    2577 generic.go:356] "Generic (PLEG): container finished" podID="49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" containerID="808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8" exitCode=0
Sep 10 15:47:03.275496 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.275444    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"
Sep 10 15:47:03.275496 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.275455    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr" event={"ID":"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc","Type":"ContainerDied","Data":"808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8"}
Sep 10 15:47:03.275671 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.275534    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr" event={"ID":"49b882b9-8e9e-4fce-a79f-23d1cc39f4bc","Type":"ContainerDied","Data":"c105d9fd69ad4fe1af580fc64020bbe0001515c09f6f2118f259a56e1fe9efa0"}
Sep 10 15:47:03.284038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.284021    2577 scope.go:117] "RemoveContainer" containerID="acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858"
Sep 10 15:47:03.284440 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:47:03.284415    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858\": container with ID starting with acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858 not found: ID does not exist" containerID="acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858"
Sep 10 15:47:03.284519 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.284450    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858"} err="failed to get container status \"acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858\": rpc error: code = NotFound desc = could not find container \"acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858\": container with ID starting with acd050795786ad9b082895085c35ad76aa652d109e6f2ecf304b49daa8e2c858 not found: ID does not exist"
Sep 10 15:47:03.284519 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.284473    2577 scope.go:117] "RemoveContainer" containerID="808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8"
Sep 10 15:47:03.294718 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.294701    2577 scope.go:117] "RemoveContainer" containerID="808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8"
Sep 10 15:47:03.295033 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:47:03.295014    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8\": container with ID starting with 808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8 not found: ID does not exist" containerID="808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8"
Sep 10 15:47:03.295100 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.295041    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8"} err="failed to get container status \"808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8\": rpc error: code = NotFound desc = could not find container \"808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8\": container with ID starting with 808dacaf6043793f394fe2e4410dbef4dc72486b7785b1a6374f872ecd4594a8 not found: ID does not exist"
Sep 10 15:47:03.298355 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.298333    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"]
Sep 10 15:47:03.303808 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.303789    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-96d7ee49-bc8f6bcdb-vv5nv"]
Sep 10 15:47:03.317127 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.317104    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"]
Sep 10 15:47:03.319411 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:03.319385    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-96d7ee49-76b8bb6f74-4vvgr"]
Sep 10 15:47:04.908806 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:04.908776    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0881ccfc-caa1-4d40-a0a2-b777d2c89654" path="/var/lib/kubelet/pods/0881ccfc-caa1-4d40-a0a2-b777d2c89654/volumes"
Sep 10 15:47:04.909196 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:04.909180    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" path="/var/lib/kubelet/pods/49b882b9-8e9e-4fce-a79f-23d1cc39f4bc/volumes"
Sep 10 15:47:07.958346 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.958314    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"]
Sep 10 15:47:07.958908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.958856    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0881ccfc-caa1-4d40-a0a2-b777d2c89654" containerName="maas-api"
Sep 10 15:47:07.958908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.958871    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="0881ccfc-caa1-4d40-a0a2-b777d2c89654" containerName="maas-api"
Sep 10 15:47:07.958908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.958896    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" containerName="payload-pre-processing"
Sep 10 15:47:07.958908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.958902    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" containerName="payload-pre-processing"
Sep 10 15:47:07.959071 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.958958    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b882b9-8e9e-4fce-a79f-23d1cc39f4bc" containerName="payload-pre-processing"
Sep 10 15:47:07.959071 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.958967    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="0881ccfc-caa1-4d40-a0a2-b777d2c89654" containerName="maas-api"
Sep 10 15:47:07.963461 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.963444    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:07.966127 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.966102    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ff2a412a-serving-cert\""
Sep 10 15:47:07.974292 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:07.974267    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"]
Sep 10 15:47:08.111461 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.111404    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv647\" (UniqueName: \"kubernetes.io/projected/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-kube-api-access-lv647\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.111652 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.111499    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-metrics-certs\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.111652 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.111535    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-openshift-service-ca\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.111652 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.111571    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-maas-api-tls\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.212580 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.212490    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-maas-api-tls\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.212771 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.212579    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv647\" (UniqueName: \"kubernetes.io/projected/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-kube-api-access-lv647\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.212771 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.212634    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-metrics-certs\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.212771 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.212665    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-openshift-service-ca\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.213438 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.213334    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-openshift-service-ca\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.215284 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.215261    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-metrics-certs\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.215284 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.215278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-maas-api-tls\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.220877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.220858    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv647\" (UniqueName: \"kubernetes.io/projected/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-kube-api-access-lv647\") pod \"maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.277456 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.277420    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:08.553653 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.553632    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"]
Sep 10 15:47:08.555478 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:47:08.555451    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c6dbd59_e589_4b36_bfe3_65ccf715abb5.slice/crio-45b901a8676d23996c66dd13e2b7555ebae3283596e6475e516d8d32fb928929 WatchSource:0}: Error finding container 45b901a8676d23996c66dd13e2b7555ebae3283596e6475e516d8d32fb928929: Status 404 returned error can't find the container with id 45b901a8676d23996c66dd13e2b7555ebae3283596e6475e516d8d32fb928929
Sep 10 15:47:08.569072 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.569047    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"]
Sep 10 15:47:08.573824 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.573803    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.576344 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.576325    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ff2a412a-dockercfg-494bw\""
Sep 10 15:47:08.576666 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.576652    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ff2a412a\""
Sep 10 15:47:08.586685 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.586661    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"]
Sep 10 15:47:08.606924 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.606899    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"]
Sep 10 15:47:08.610396 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.610376    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:08.620174 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.620153    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"]
Sep 10 15:47:08.716698 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.716669    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clnwn\" (UniqueName: \"kubernetes.io/projected/6a045bad-14fe-47a3-ac1b-7eace1e287e2-kube-api-access-clnwn\") pod \"payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") " pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:08.716893 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.716710    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa29ac86-4f71-4e71-a762-a90e9b00d90f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.716893 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.716815    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a045bad-14fe-47a3-ac1b-7eace1e287e2-plugins-config-volume\") pod \"payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") " pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:08.716893 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.716854    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wglhp\" (UniqueName: \"kubernetes.io/projected/aa29ac86-4f71-4e71-a762-a90e9b00d90f-kube-api-access-wglhp\") pod \"payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.818251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.818162    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clnwn\" (UniqueName: \"kubernetes.io/projected/6a045bad-14fe-47a3-ac1b-7eace1e287e2-kube-api-access-clnwn\") pod \"payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") " pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:08.818251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.818236    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa29ac86-4f71-4e71-a762-a90e9b00d90f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.818509 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.818291    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a045bad-14fe-47a3-ac1b-7eace1e287e2-plugins-config-volume\") pod \"payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") " pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:08.818509 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.818330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wglhp\" (UniqueName: \"kubernetes.io/projected/aa29ac86-4f71-4e71-a762-a90e9b00d90f-kube-api-access-wglhp\") pod \"payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.819015 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.818986    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa29ac86-4f71-4e71-a762-a90e9b00d90f-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.819132 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.819112    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a045bad-14fe-47a3-ac1b-7eace1e287e2-plugins-config-volume\") pod \"payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") " pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:08.826317 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.826289    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wglhp\" (UniqueName: \"kubernetes.io/projected/aa29ac86-4f71-4e71-a762-a90e9b00d90f-kube-api-access-wglhp\") pod \"payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.826427 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.826342    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-clnwn\" (UniqueName: \"kubernetes.io/projected/6a045bad-14fe-47a3-ac1b-7eace1e287e2-kube-api-access-clnwn\") pod \"payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") " pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:08.885396 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.885357    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:08.924348 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:08.924317    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:09.162873 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.162849    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"]
Sep 10 15:47:09.164359 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:47:09.164329    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa29ac86_4f71_4e71_a762_a90e9b00d90f.slice/crio-d2cc255cb58841fd616c1a0c6f4e97460053811c89d156615d50438c377f2a1c WatchSource:0}: Error finding container d2cc255cb58841fd616c1a0c6f4e97460053811c89d156615d50438c377f2a1c: Status 404 returned error can't find the container with id d2cc255cb58841fd616c1a0c6f4e97460053811c89d156615d50438c377f2a1c
Sep 10 15:47:09.204897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.204874    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"]
Sep 10 15:47:09.208840 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:47:09.208809    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a045bad_14fe_47a3_ac1b_7eace1e287e2.slice/crio-65021dbbd832d65984e6f0e6bc61b64ff142376bf04a9ba0608e71a50daedf37 WatchSource:0}: Error finding container 65021dbbd832d65984e6f0e6bc61b64ff142376bf04a9ba0608e71a50daedf37: Status 404 returned error can't find the container with id 65021dbbd832d65984e6f0e6bc61b64ff142376bf04a9ba0608e71a50daedf37
Sep 10 15:47:09.322244 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.322208    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt" event={"ID":"aa29ac86-4f71-4e71-a762-a90e9b00d90f","Type":"ContainerStarted","Data":"3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d"}
Sep 10 15:47:09.322418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.322268    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt" event={"ID":"aa29ac86-4f71-4e71-a762-a90e9b00d90f","Type":"ContainerStarted","Data":"d2cc255cb58841fd616c1a0c6f4e97460053811c89d156615d50438c377f2a1c"}
Sep 10 15:47:09.322418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.322296    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:09.324115 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.324088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk" event={"ID":"6a045bad-14fe-47a3-ac1b-7eace1e287e2","Type":"ContainerStarted","Data":"7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de"}
Sep 10 15:47:09.324251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.324144    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk" event={"ID":"6a045bad-14fe-47a3-ac1b-7eace1e287e2","Type":"ContainerStarted","Data":"65021dbbd832d65984e6f0e6bc61b64ff142376bf04a9ba0608e71a50daedf37"}
Sep 10 15:47:09.324251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.324175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:09.325943 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.325909    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp" event={"ID":"3c6dbd59-e589-4b36-bfe3-65ccf715abb5","Type":"ContainerStarted","Data":"b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347"}
Sep 10 15:47:09.326054 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.325955    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp" event={"ID":"3c6dbd59-e589-4b36-bfe3-65ccf715abb5","Type":"ContainerStarted","Data":"45b901a8676d23996c66dd13e2b7555ebae3283596e6475e516d8d32fb928929"}
Sep 10 15:47:09.326054 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.325991    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:09.338077 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.338028    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt" podStartSLOduration=1.338012762 podStartE2EDuration="1.338012762s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:09.336590224 +0000 UTC m=+872.950717136" watchObservedRunningTime="2026-09-10 15:47:09.338012762 +0000 UTC m=+872.952139702"
Sep 10 15:47:09.352821 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.352771    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk" podStartSLOduration=1.352739611 podStartE2EDuration="1.352739611s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:09.351716061 +0000 UTC m=+872.965842966" watchObservedRunningTime="2026-09-10 15:47:09.352739611 +0000 UTC m=+872.966866515"
Sep 10 15:47:09.369428 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:09.369377    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp" podStartSLOduration=1.881343324 podStartE2EDuration="2.369360548s" totalImagesPullingTime="488.017224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:09.366458479 +0000 UTC m=+872.980585393" watchObservedRunningTime="2026-09-10 15:47:09.369360548 +0000 UTC m=+872.983487451"
Sep 10 15:47:15.337578 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:15.337549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:20.333577 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:20.333547    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:20.334097 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:20.333624    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:33.868885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:33.868797    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"]
Sep 10 15:47:33.869353 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:33.869076    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz" podUID="a11c9993-90d3-421f-b20b-564ebcf1784f" containerName="payload-processing" containerID="cri-o://e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed" gracePeriod=30
Sep 10 15:47:33.876440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:33.876406    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"]
Sep 10 15:47:33.876681 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:33.876636    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx" podUID="06d11f33-2408-48e8-a2fd-05bbb98ba3f9" containerName="payload-pre-processing" containerID="cri-o://f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4" gracePeriod=30
Sep 10 15:47:34.158397 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.158377    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:47:34.161151 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.161133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:47:34.258989 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.258956    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjd2j\" (UniqueName: \"kubernetes.io/projected/a11c9993-90d3-421f-b20b-564ebcf1784f-kube-api-access-rjd2j\") pod \"a11c9993-90d3-421f-b20b-564ebcf1784f\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") "
Sep 10 15:47:34.258989 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.258995    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-plugins-config-volume\") pod \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") "
Sep 10 15:47:34.259202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.259070    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11c9993-90d3-421f-b20b-564ebcf1784f-plugins-config-volume\") pod \"a11c9993-90d3-421f-b20b-564ebcf1784f\" (UID: \"a11c9993-90d3-421f-b20b-564ebcf1784f\") "
Sep 10 15:47:34.259202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.259086    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5zkbk\" (UniqueName: \"kubernetes.io/projected/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-kube-api-access-5zkbk\") pod \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\" (UID: \"06d11f33-2408-48e8-a2fd-05bbb98ba3f9\") "
Sep 10 15:47:34.259457 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.259433    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a11c9993-90d3-421f-b20b-564ebcf1784f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a11c9993-90d3-421f-b20b-564ebcf1784f" (UID: "a11c9993-90d3-421f-b20b-564ebcf1784f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:47:34.259522 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.259473    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06d11f33-2408-48e8-a2fd-05bbb98ba3f9" (UID: "06d11f33-2408-48e8-a2fd-05bbb98ba3f9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:47:34.261293 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.261268    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a11c9993-90d3-421f-b20b-564ebcf1784f-kube-api-access-rjd2j" (OuterVolumeSpecName: "kube-api-access-rjd2j") pod "a11c9993-90d3-421f-b20b-564ebcf1784f" (UID: "a11c9993-90d3-421f-b20b-564ebcf1784f"). InnerVolumeSpecName "kube-api-access-rjd2j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:34.261413 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.261295    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-kube-api-access-5zkbk" (OuterVolumeSpecName: "kube-api-access-5zkbk") pod "06d11f33-2408-48e8-a2fd-05bbb98ba3f9" (UID: "06d11f33-2408-48e8-a2fd-05bbb98ba3f9"). InnerVolumeSpecName "kube-api-access-5zkbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:34.359989 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.359957    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a11c9993-90d3-421f-b20b-564ebcf1784f-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:34.359989 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.359984    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5zkbk\" (UniqueName: \"kubernetes.io/projected/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-kube-api-access-5zkbk\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:34.359989 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.359994    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjd2j\" (UniqueName: \"kubernetes.io/projected/a11c9993-90d3-421f-b20b-564ebcf1784f-kube-api-access-rjd2j\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:34.360227 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.360004    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d11f33-2408-48e8-a2fd-05bbb98ba3f9-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:34.469268 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.469174    2577 generic.go:356] "Generic (PLEG): container finished" podID="a11c9993-90d3-421f-b20b-564ebcf1784f" containerID="e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed" exitCode=0
Sep 10 15:47:34.469268 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.469224    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"
Sep 10 15:47:34.469268 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.469257    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz" event={"ID":"a11c9993-90d3-421f-b20b-564ebcf1784f","Type":"ContainerDied","Data":"e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed"}
Sep 10 15:47:34.469548 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.469316    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz" event={"ID":"a11c9993-90d3-421f-b20b-564ebcf1784f","Type":"ContainerDied","Data":"e8b29a2636195db2088b1cdfd2a01df3b8a4a0c1a9c7edd45ee695f13dc228a1"}
Sep 10 15:47:34.469548 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.469344    2577 scope.go:117] "RemoveContainer" containerID="e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed"
Sep 10 15:47:34.470781 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.470743    2577 generic.go:356] "Generic (PLEG): container finished" podID="06d11f33-2408-48e8-a2fd-05bbb98ba3f9" containerID="f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4" exitCode=0
Sep 10 15:47:34.470871 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.470809    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"
Sep 10 15:47:34.470871 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.470842    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx" event={"ID":"06d11f33-2408-48e8-a2fd-05bbb98ba3f9","Type":"ContainerDied","Data":"f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4"}
Sep 10 15:47:34.470983 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.470880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx" event={"ID":"06d11f33-2408-48e8-a2fd-05bbb98ba3f9","Type":"ContainerDied","Data":"b062e8e04faa1f792f1aec1d12462c020b1d707d8f6abc9ec99c7cd75207d051"}
Sep 10 15:47:34.481116 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.481078    2577 scope.go:117] "RemoveContainer" containerID="e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed"
Sep 10 15:47:34.481983 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:47:34.481905    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed\": container with ID starting with e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed not found: ID does not exist" containerID="e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed"
Sep 10 15:47:34.481983 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.481943    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed"} err="failed to get container status \"e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed\": rpc error: code = NotFound desc = could not find container \"e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed\": container with ID starting with e7c4b369ea8f941604ba68748ad86a6af61b51501ff8efa6e89d7f27663c62ed not found: ID does not exist"
Sep 10 15:47:34.481983 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.481967    2577 scope.go:117] "RemoveContainer" containerID="f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4"
Sep 10 15:47:34.491883 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.491865    2577 scope.go:117] "RemoveContainer" containerID="f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4"
Sep 10 15:47:34.492221 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:47:34.492197    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4\": container with ID starting with f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4 not found: ID does not exist" containerID="f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4"
Sep 10 15:47:34.492280 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.492233    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4"} err="failed to get container status \"f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4\": rpc error: code = NotFound desc = could not find container \"f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4\": container with ID starting with f775e128ad5fe44bc1567e4c923fdaf1993514c82d4503ecf0c367917c049fc4 not found: ID does not exist"
Sep 10 15:47:34.495926 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.495904    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"]
Sep 10 15:47:34.499501 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.499481    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-d5048f-85dbbf544b-kgptx"]
Sep 10 15:47:34.512229 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.512209    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"]
Sep 10 15:47:34.515877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.515857    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-d5048f-d4c6b8867-9kbpz"]
Sep 10 15:47:34.909703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.909667    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d11f33-2408-48e8-a2fd-05bbb98ba3f9" path="/var/lib/kubelet/pods/06d11f33-2408-48e8-a2fd-05bbb98ba3f9/volumes"
Sep 10 15:47:34.910093 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:34.910078    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a11c9993-90d3-421f-b20b-564ebcf1784f" path="/var/lib/kubelet/pods/a11c9993-90d3-421f-b20b-564ebcf1784f/volumes"
Sep 10 15:47:38.883195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:38.883159    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"]
Sep 10 15:47:38.883604 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:38.883359    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp" podUID="3c6dbd59-e589-4b36-bfe3-65ccf715abb5" containerName="maas-api" containerID="cri-o://b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347" gracePeriod=30
Sep 10 15:47:38.977917 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:38.977886    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"]
Sep 10 15:47:38.978134 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:38.978110    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk" podUID="6a045bad-14fe-47a3-ac1b-7eace1e287e2" containerName="payload-processing" containerID="cri-o://7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de" gracePeriod=30
Sep 10 15:47:38.986442 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:38.986060    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"]
Sep 10 15:47:38.986442 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:38.986279    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt" podUID="aa29ac86-4f71-4e71-a762-a90e9b00d90f" containerName="payload-pre-processing" containerID="cri-o://3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d" gracePeriod=30
Sep 10 15:47:39.168051 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.168025    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:39.236067 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.236039    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:39.282709 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.282684    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:39.304812 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.304780    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-clnwn\" (UniqueName: \"kubernetes.io/projected/6a045bad-14fe-47a3-ac1b-7eace1e287e2-kube-api-access-clnwn\") pod \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") "
Sep 10 15:47:39.304999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.304863    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lv647\" (UniqueName: \"kubernetes.io/projected/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-kube-api-access-lv647\") pod \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") "
Sep 10 15:47:39.305079 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.305056    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-metrics-certs\") pod \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") "
Sep 10 15:47:39.305140 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.305126    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-openshift-service-ca\") pod \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") "
Sep 10 15:47:39.305269 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.305232    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a045bad-14fe-47a3-ac1b-7eace1e287e2-plugins-config-volume\") pod \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\" (UID: \"6a045bad-14fe-47a3-ac1b-7eace1e287e2\") "
Sep 10 15:47:39.305404 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.305304    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-maas-api-tls\") pod \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\" (UID: \"3c6dbd59-e589-4b36-bfe3-65ccf715abb5\") "
Sep 10 15:47:39.306275 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.306243    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c6dbd59-e589-4b36-bfe3-65ccf715abb5" (UID: "3c6dbd59-e589-4b36-bfe3-65ccf715abb5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:47:39.306380 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.306328    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a045bad-14fe-47a3-ac1b-7eace1e287e2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6a045bad-14fe-47a3-ac1b-7eace1e287e2" (UID: "6a045bad-14fe-47a3-ac1b-7eace1e287e2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:47:39.307841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.307716    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-kube-api-access-lv647" (OuterVolumeSpecName: "kube-api-access-lv647") pod "3c6dbd59-e589-4b36-bfe3-65ccf715abb5" (UID: "3c6dbd59-e589-4b36-bfe3-65ccf715abb5"). InnerVolumeSpecName "kube-api-access-lv647". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:39.307981 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.307956    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a045bad-14fe-47a3-ac1b-7eace1e287e2-kube-api-access-clnwn" (OuterVolumeSpecName: "kube-api-access-clnwn") pod "6a045bad-14fe-47a3-ac1b-7eace1e287e2" (UID: "6a045bad-14fe-47a3-ac1b-7eace1e287e2"). InnerVolumeSpecName "kube-api-access-clnwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:39.308603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.308569    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c6dbd59-e589-4b36-bfe3-65ccf715abb5" (UID: "3c6dbd59-e589-4b36-bfe3-65ccf715abb5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:47:39.309072 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.309036    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3c6dbd59-e589-4b36-bfe3-65ccf715abb5" (UID: "3c6dbd59-e589-4b36-bfe3-65ccf715abb5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:47:39.406568 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406487    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wglhp\" (UniqueName: \"kubernetes.io/projected/aa29ac86-4f71-4e71-a762-a90e9b00d90f-kube-api-access-wglhp\") pod \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") "
Sep 10 15:47:39.406568 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406527    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa29ac86-4f71-4e71-a762-a90e9b00d90f-plugins-config-volume\") pod \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\" (UID: \"aa29ac86-4f71-4e71-a762-a90e9b00d90f\") "
Sep 10 15:47:39.406791 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406707    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6a045bad-14fe-47a3-ac1b-7eace1e287e2-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.406791 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406729    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.406791 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406743    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clnwn\" (UniqueName: \"kubernetes.io/projected/6a045bad-14fe-47a3-ac1b-7eace1e287e2-kube-api-access-clnwn\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.406897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406789    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv647\" (UniqueName: \"kubernetes.io/projected/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-kube-api-access-lv647\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.406897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406803    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.406897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406817    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c6dbd59-e589-4b36-bfe3-65ccf715abb5-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.406995 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.406979    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa29ac86-4f71-4e71-a762-a90e9b00d90f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aa29ac86-4f71-4e71-a762-a90e9b00d90f" (UID: "aa29ac86-4f71-4e71-a762-a90e9b00d90f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:47:39.408691 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.408669    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa29ac86-4f71-4e71-a762-a90e9b00d90f-kube-api-access-wglhp" (OuterVolumeSpecName: "kube-api-access-wglhp") pod "aa29ac86-4f71-4e71-a762-a90e9b00d90f" (UID: "aa29ac86-4f71-4e71-a762-a90e9b00d90f"). InnerVolumeSpecName "kube-api-access-wglhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:47:39.503057 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.503019    2577 generic.go:356] "Generic (PLEG): container finished" podID="3c6dbd59-e589-4b36-bfe3-65ccf715abb5" containerID="b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347" exitCode=0
Sep 10 15:47:39.503240 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.503065    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"
Sep 10 15:47:39.503240 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.503105    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp" event={"ID":"3c6dbd59-e589-4b36-bfe3-65ccf715abb5","Type":"ContainerDied","Data":"b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347"}
Sep 10 15:47:39.503240 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.503179    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp" event={"ID":"3c6dbd59-e589-4b36-bfe3-65ccf715abb5","Type":"ContainerDied","Data":"45b901a8676d23996c66dd13e2b7555ebae3283596e6475e516d8d32fb928929"}
Sep 10 15:47:39.503240 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.503213    2577 scope.go:117] "RemoveContainer" containerID="b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347"
Sep 10 15:47:39.504661 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.504640    2577 generic.go:356] "Generic (PLEG): container finished" podID="aa29ac86-4f71-4e71-a762-a90e9b00d90f" containerID="3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d" exitCode=0
Sep 10 15:47:39.504795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.504693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"
Sep 10 15:47:39.504795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.504787    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt" event={"ID":"aa29ac86-4f71-4e71-a762-a90e9b00d90f","Type":"ContainerDied","Data":"3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d"}
Sep 10 15:47:39.504919 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.504827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt" event={"ID":"aa29ac86-4f71-4e71-a762-a90e9b00d90f","Type":"ContainerDied","Data":"d2cc255cb58841fd616c1a0c6f4e97460053811c89d156615d50438c377f2a1c"}
Sep 10 15:47:39.506377 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.506353    2577 generic.go:356] "Generic (PLEG): container finished" podID="6a045bad-14fe-47a3-ac1b-7eace1e287e2" containerID="7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de" exitCode=0
Sep 10 15:47:39.506503 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.506395    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"
Sep 10 15:47:39.506503 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.506398    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk" event={"ID":"6a045bad-14fe-47a3-ac1b-7eace1e287e2","Type":"ContainerDied","Data":"7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de"}
Sep 10 15:47:39.506503 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.506437    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk" event={"ID":"6a045bad-14fe-47a3-ac1b-7eace1e287e2","Type":"ContainerDied","Data":"65021dbbd832d65984e6f0e6bc61b64ff142376bf04a9ba0608e71a50daedf37"}
Sep 10 15:47:39.507183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.507155    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wglhp\" (UniqueName: \"kubernetes.io/projected/aa29ac86-4f71-4e71-a762-a90e9b00d90f-kube-api-access-wglhp\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.507183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.507177    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa29ac86-4f71-4e71-a762-a90e9b00d90f-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:47:39.512940 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.512924    2577 scope.go:117] "RemoveContainer" containerID="b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347"
Sep 10 15:47:39.513287 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:47:39.513251    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347\": container with ID starting with b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347 not found: ID does not exist" containerID="b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347"
Sep 10 15:47:39.513399 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.513286    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347"} err="failed to get container status \"b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347\": rpc error: code = NotFound desc = could not find container \"b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347\": container with ID starting with b0045ad34f828df1bd37e2eab456825c6dd9cce34bca65d63447df47832b6347 not found: ID does not exist"
Sep 10 15:47:39.513399 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.513308    2577 scope.go:117] "RemoveContainer" containerID="3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d"
Sep 10 15:47:39.522850 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.522823    2577 scope.go:117] "RemoveContainer" containerID="3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d"
Sep 10 15:47:39.523128 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:47:39.523112    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d\": container with ID starting with 3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d not found: ID does not exist" containerID="3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d"
Sep 10 15:47:39.523202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.523137    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d"} err="failed to get container status \"3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d\": rpc error: code = NotFound desc = could not find container \"3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d\": container with ID starting with 3b62c1b077635c76e326b550ac075db57be5d886abddf337f1ebc275134cbc0d not found: ID does not exist"
Sep 10 15:47:39.523202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.523158    2577 scope.go:117] "RemoveContainer" containerID="7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de"
Sep 10 15:47:39.534387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.534367    2577 scope.go:117] "RemoveContainer" containerID="7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de"
Sep 10 15:47:39.534459 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.534428    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"]
Sep 10 15:47:39.534712 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:47:39.534692    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de\": container with ID starting with 7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de not found: ID does not exist" containerID="7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de"
Sep 10 15:47:39.534825 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.534741    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de"} err="failed to get container status \"7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de\": rpc error: code = NotFound desc = could not find container \"7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de\": container with ID starting with 7594944f98d79d42774d3a90e4f348d034981be4da005a7a133a4f6ae8a4f5de not found: ID does not exist"
Sep 10 15:47:39.536756 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.536730    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ff2a412a-798688d9f-9nfmp"]
Sep 10 15:47:39.543861 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.543742    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"]
Sep 10 15:47:39.547526 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.547508    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-ff2a412a-56678948cc-m59rk"]
Sep 10 15:47:39.556680 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.556655    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"]
Sep 10 15:47:39.560251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:39.560233    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ff2a412a-56b4cc59f4-wdkdt"]
Sep 10 15:47:40.908198 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:40.908164    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c6dbd59-e589-4b36-bfe3-65ccf715abb5" path="/var/lib/kubelet/pods/3c6dbd59-e589-4b36-bfe3-65ccf715abb5/volumes"
Sep 10 15:47:40.908806 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:40.908510    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a045bad-14fe-47a3-ac1b-7eace1e287e2" path="/var/lib/kubelet/pods/6a045bad-14fe-47a3-ac1b-7eace1e287e2/volumes"
Sep 10 15:47:40.908885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:40.908824    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa29ac86-4f71-4e71-a762-a90e9b00d90f" path="/var/lib/kubelet/pods/aa29ac86-4f71-4e71-a762-a90e9b00d90f/volumes"
Sep 10 15:47:41.440797 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.440763    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"]
Sep 10 15:47:41.441145 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441132    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d11f33-2408-48e8-a2fd-05bbb98ba3f9" containerName="payload-pre-processing"
Sep 10 15:47:41.441195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441147    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d11f33-2408-48e8-a2fd-05bbb98ba3f9" containerName="payload-pre-processing"
Sep 10 15:47:41.441195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441160    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a11c9993-90d3-421f-b20b-564ebcf1784f" containerName="payload-processing"
Sep 10 15:47:41.441195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441165    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a11c9993-90d3-421f-b20b-564ebcf1784f" containerName="payload-processing"
Sep 10 15:47:41.441195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441176    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c6dbd59-e589-4b36-bfe3-65ccf715abb5" containerName="maas-api"
Sep 10 15:47:41.441195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441183    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c6dbd59-e589-4b36-bfe3-65ccf715abb5" containerName="maas-api"
Sep 10 15:47:41.441195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441188    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a045bad-14fe-47a3-ac1b-7eace1e287e2" containerName="payload-processing"
Sep 10 15:47:41.441195 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441193    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a045bad-14fe-47a3-ac1b-7eace1e287e2" containerName="payload-processing"
Sep 10 15:47:41.441406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441204    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa29ac86-4f71-4e71-a762-a90e9b00d90f" containerName="payload-pre-processing"
Sep 10 15:47:41.441406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441209    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa29ac86-4f71-4e71-a762-a90e9b00d90f" containerName="payload-pre-processing"
Sep 10 15:47:41.441406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441260    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa29ac86-4f71-4e71-a762-a90e9b00d90f" containerName="payload-pre-processing"
Sep 10 15:47:41.441406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441269    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a11c9993-90d3-421f-b20b-564ebcf1784f" containerName="payload-processing"
Sep 10 15:47:41.441406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441276    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d11f33-2408-48e8-a2fd-05bbb98ba3f9" containerName="payload-pre-processing"
Sep 10 15:47:41.441406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441283    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c6dbd59-e589-4b36-bfe3-65ccf715abb5" containerName="maas-api"
Sep 10 15:47:41.441406 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.441289    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a045bad-14fe-47a3-ac1b-7eace1e287e2" containerName="payload-processing"
Sep 10 15:47:41.445806 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.445788    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.448988 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.448964    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80d47c\"/\"default-dockercfg-4g8sl\""
Sep 10 15:47:41.448988 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.448978    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-80d47c\"/\"test-model-80d47c-kserve-self-signed-certs\""
Sep 10 15:47:41.449177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.448965    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80d47c\"/\"openshift-service-ca.crt\""
Sep 10 15:47:41.449177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.449024    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-80d47c\"/\"kube-root-ca.crt\""
Sep 10 15:47:41.455248 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.455221    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"]
Sep 10 15:47:41.525400 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.525368    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-home\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.525400 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.525402    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-dshm\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.525625 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.525428    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-tmp-dir\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.525625 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.525490    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbf2k\" (UniqueName: \"kubernetes.io/projected/583e1afb-2da8-4c39-920d-3db301f63b55-kube-api-access-kbf2k\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.525625 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.525528    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e1afb-2da8-4c39-920d-3db301f63b55-tls-certs\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.525625 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.525553    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-model-cache\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.626830 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.626789    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-dshm\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.626842    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-tmp-dir\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.626880    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbf2k\" (UniqueName: \"kubernetes.io/projected/583e1afb-2da8-4c39-920d-3db301f63b55-kube-api-access-kbf2k\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.627017    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e1afb-2da8-4c39-920d-3db301f63b55-tls-certs\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.627079    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-model-cache\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627224 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.627145    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-home\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627302 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.627278    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-tmp-dir\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627480 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.627456    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-model-cache\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.627608 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.627464    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-home\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.629368 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.629339    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-dshm\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.629649 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.629629    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e1afb-2da8-4c39-920d-3db301f63b55-tls-certs\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.634822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.634800    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbf2k\" (UniqueName: \"kubernetes.io/projected/583e1afb-2da8-4c39-920d-3db301f63b55-kube-api-access-kbf2k\") pod \"test-model-80d47c-kserve-748496b457-4vxn5\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") " pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:41.756215 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:41.756125    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:42.028064 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:42.028038    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"]
Sep 10 15:47:42.029949 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:47:42.029922    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod583e1afb_2da8_4c39_920d_3db301f63b55.slice/crio-16a09f5cd6ee769de601f779e6d5b583921f258d9f07685cff5003f9372071cf WatchSource:0}: Error finding container 16a09f5cd6ee769de601f779e6d5b583921f258d9f07685cff5003f9372071cf: Status 404 returned error can't find the container with id 16a09f5cd6ee769de601f779e6d5b583921f258d9f07685cff5003f9372071cf
Sep 10 15:47:42.527147 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:42.527112    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5" event={"ID":"583e1afb-2da8-4c39-920d-3db301f63b55","Type":"ContainerStarted","Data":"2ec7b353dd7afcff806ba28dedfb2fdd92978de3c61537876abce4d69b53f3f1"}
Sep 10 15:47:42.527147 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:42.527159    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5" event={"ID":"583e1afb-2da8-4c39-920d-3db301f63b55","Type":"ContainerStarted","Data":"16a09f5cd6ee769de601f779e6d5b583921f258d9f07685cff5003f9372071cf"}
Sep 10 15:47:42.544641 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:42.544597    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5" podStartSLOduration=1.544581937 podStartE2EDuration="1.544581937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:42.541891168 +0000 UTC m=+906.156018072" watchObservedRunningTime="2026-09-10 15:47:42.544581937 +0000 UTC m=+906.158708841"
Sep 10 15:47:51.756367 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:51.756332    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:51.756784 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:51.756497    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:51.768617 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:51.768595    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:47:52.593249 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:47:52.593221    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:48:22.209109 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.209072    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"]
Sep 10 15:48:22.212608 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.212579    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.215746 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.215722    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-76a9b8\""
Sep 10 15:48:22.215893 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.215817    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-76a9b8-dockercfg-pzz2r\""
Sep 10 15:48:22.225315 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.225292    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"]
Sep 10 15:48:22.277245 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.277214    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vb8mj\" (UniqueName: \"kubernetes.io/projected/14c12a3f-9a73-48fe-8467-c29a2975d160-kube-api-access-vb8mj\") pod \"payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.277402 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.277274    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c12a3f-9a73-48fe-8467-c29a2975d160-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.378700 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.378653    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vb8mj\" (UniqueName: \"kubernetes.io/projected/14c12a3f-9a73-48fe-8467-c29a2975d160-kube-api-access-vb8mj\") pod \"payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.378906 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.378716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c12a3f-9a73-48fe-8467-c29a2975d160-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.379348 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.379329    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c12a3f-9a73-48fe-8467-c29a2975d160-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.386565 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.386535    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vb8mj\" (UniqueName: \"kubernetes.io/projected/14c12a3f-9a73-48fe-8467-c29a2975d160-kube-api-access-vb8mj\") pod \"payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.525239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.525152    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:22.802430 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:22.802400    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"]
Sep 10 15:48:22.803521 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:48:22.803498    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14c12a3f_9a73_48fe_8467_c29a2975d160.slice/crio-176c52750c41aebd2c3f12cddcd646bfb91e1bce7ca5ed8a8f768ee14f1cabb9 WatchSource:0}: Error finding container 176c52750c41aebd2c3f12cddcd646bfb91e1bce7ca5ed8a8f768ee14f1cabb9: Status 404 returned error can't find the container with id 176c52750c41aebd2c3f12cddcd646bfb91e1bce7ca5ed8a8f768ee14f1cabb9
Sep 10 15:48:23.755232 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:23.755196    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk" event={"ID":"14c12a3f-9a73-48fe-8467-c29a2975d160","Type":"ContainerStarted","Data":"272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def"}
Sep 10 15:48:23.755773 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:23.755254    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk" event={"ID":"14c12a3f-9a73-48fe-8467-c29a2975d160","Type":"ContainerStarted","Data":"176c52750c41aebd2c3f12cddcd646bfb91e1bce7ca5ed8a8f768ee14f1cabb9"}
Sep 10 15:48:23.755773 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:23.755312    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:48:23.771923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:23.771878    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk" podStartSLOduration=1.771866922 podStartE2EDuration="1.771866922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:48:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:48:23.770496991 +0000 UTC m=+947.384623895" watchObservedRunningTime="2026-09-10 15:48:23.771866922 +0000 UTC m=+947.385993826"
Sep 10 15:48:34.763702 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:48:34.763674    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:49:03.217293 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.217201    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"]
Sep 10 15:49:03.217841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.217493    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk" podUID="14c12a3f-9a73-48fe-8467-c29a2975d160" containerName="payload-pre-processing" containerID="cri-o://272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def" gracePeriod=30
Sep 10 15:49:03.478028 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.477970    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:49:03.538855 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.538819    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vb8mj\" (UniqueName: \"kubernetes.io/projected/14c12a3f-9a73-48fe-8467-c29a2975d160-kube-api-access-vb8mj\") pod \"14c12a3f-9a73-48fe-8467-c29a2975d160\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") "
Sep 10 15:49:03.538855 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.538858    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c12a3f-9a73-48fe-8467-c29a2975d160-plugins-config-volume\") pod \"14c12a3f-9a73-48fe-8467-c29a2975d160\" (UID: \"14c12a3f-9a73-48fe-8467-c29a2975d160\") "
Sep 10 15:49:03.539287 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.539262    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14c12a3f-9a73-48fe-8467-c29a2975d160-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "14c12a3f-9a73-48fe-8467-c29a2975d160" (UID: "14c12a3f-9a73-48fe-8467-c29a2975d160"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:49:03.541185 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.541155    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14c12a3f-9a73-48fe-8467-c29a2975d160-kube-api-access-vb8mj" (OuterVolumeSpecName: "kube-api-access-vb8mj") pod "14c12a3f-9a73-48fe-8467-c29a2975d160" (UID: "14c12a3f-9a73-48fe-8467-c29a2975d160"). InnerVolumeSpecName "kube-api-access-vb8mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:49:03.640333 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.640301    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vb8mj\" (UniqueName: \"kubernetes.io/projected/14c12a3f-9a73-48fe-8467-c29a2975d160-kube-api-access-vb8mj\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:49:03.640333 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.640330    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14c12a3f-9a73-48fe-8467-c29a2975d160-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:49:03.986734 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.986698    2577 generic.go:356] "Generic (PLEG): container finished" podID="14c12a3f-9a73-48fe-8467-c29a2975d160" containerID="272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def" exitCode=0
Sep 10 15:49:03.986941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.986773    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"
Sep 10 15:49:03.986941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.986788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk" event={"ID":"14c12a3f-9a73-48fe-8467-c29a2975d160","Type":"ContainerDied","Data":"272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def"}
Sep 10 15:49:03.986941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.986847    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk" event={"ID":"14c12a3f-9a73-48fe-8467-c29a2975d160","Type":"ContainerDied","Data":"176c52750c41aebd2c3f12cddcd646bfb91e1bce7ca5ed8a8f768ee14f1cabb9"}
Sep 10 15:49:03.986941 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.986869    2577 scope.go:117] "RemoveContainer" containerID="272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def"
Sep 10 15:49:03.998089 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.998058    2577 scope.go:117] "RemoveContainer" containerID="272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def"
Sep 10 15:49:03.998410 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:49:03.998381    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def\": container with ID starting with 272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def not found: ID does not exist" containerID="272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def"
Sep 10 15:49:03.998499 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:03.998416    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def"} err="failed to get container status \"272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def\": rpc error: code = NotFound desc = could not find container \"272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def\": container with ID starting with 272b78e04d5f76908d86e5a527727372537e3396f50e90d912d87fa261463def not found: ID does not exist"
Sep 10 15:49:04.010439 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:04.010419    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"]
Sep 10 15:49:04.014882 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:04.014861    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-76a9b8-59595b46f8-frgmk"]
Sep 10 15:49:04.912819 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:04.912787    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="14c12a3f-9a73-48fe-8467-c29a2975d160" path="/var/lib/kubelet/pods/14c12a3f-9a73-48fe-8467-c29a2975d160/volumes"
Sep 10 15:49:18.943219 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:18.943186    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"]
Sep 10 15:49:18.943627 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:18.943583    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="14c12a3f-9a73-48fe-8467-c29a2975d160" containerName="payload-pre-processing"
Sep 10 15:49:18.943627 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:18.943596    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="14c12a3f-9a73-48fe-8467-c29a2975d160" containerName="payload-pre-processing"
Sep 10 15:49:18.943700 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:18.943660    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="14c12a3f-9a73-48fe-8467-c29a2975d160" containerName="payload-pre-processing"
Sep 10 15:49:18.947928 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:18.947908    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:18.950988 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:18.950964    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-25381b71-serving-cert\""
Sep 10 15:49:18.957043 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:18.956894    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"]
Sep 10 15:49:19.072954 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.072920    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8a0a86-b576-47d1-a9e3-4cf862287562-openshift-service-ca\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.072954 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.072954    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-metrics-certs\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.073201 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.073053    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.073201 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.073092    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fb64\" (UniqueName: \"kubernetes.io/projected/6e8a0a86-b576-47d1-a9e3-4cf862287562-kube-api-access-9fb64\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.174059 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.174021    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.174266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.174064    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fb64\" (UniqueName: \"kubernetes.io/projected/6e8a0a86-b576-47d1-a9e3-4cf862287562-kube-api-access-9fb64\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.174266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.174111    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8a0a86-b576-47d1-a9e3-4cf862287562-openshift-service-ca\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.174266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.174136    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-metrics-certs\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.174266 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:49:19.174184    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-serving-cert: secret "maas-api-e2e-mt-25381b71-serving-cert" not found
Sep 10 15:49:19.174266 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:49:19.174248    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls podName:6e8a0a86-b576-47d1-a9e3-4cf862287562 nodeName:}" failed. No retries permitted until 2026-09-10 15:49:19.674227052 +0000 UTC m=+1003.288353953 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls") pod "maas-api-e2e-mt-25381b71-97b79bbc5-2969x" (UID: "6e8a0a86-b576-47d1-a9e3-4cf862287562") : secret "maas-api-e2e-mt-25381b71-serving-cert" not found
Sep 10 15:49:19.174822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.174798    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8a0a86-b576-47d1-a9e3-4cf862287562-openshift-service-ca\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.176658 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.176636    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-metrics-certs\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.183166 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.183143    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fb64\" (UniqueName: \"kubernetes.io/projected/6e8a0a86-b576-47d1-a9e3-4cf862287562-kube-api-access-9fb64\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.578962 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.578929    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"]
Sep 10 15:49:19.582703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.582679    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.585651 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.585632    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-25381b71-dockercfg-v6gwp\""
Sep 10 15:49:19.587600 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.587580    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-25381b71\""
Sep 10 15:49:19.597310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.597277    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"]
Sep 10 15:49:19.610158 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.610117    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"]
Sep 10 15:49:19.614102 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.614081    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:19.623859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.623833    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"]
Sep 10 15:49:19.678283 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.678253    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164de106-8245-4902-a61c-700304d5039c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.678471 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.678297    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr59b\" (UniqueName: \"kubernetes.io/projected/5b3c5409-70da-4caf-8d37-515583d6b7a5-kube-api-access-qr59b\") pod \"payload-processing-e2e-mt-25381b71-5866f6986f-x77xg\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") " pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:19.678471 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.678355    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.678471 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.678374    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twg28\" (UniqueName: \"kubernetes.io/projected/164de106-8245-4902-a61c-700304d5039c-kube-api-access-twg28\") pod \"payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.678471 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.678395    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3c5409-70da-4caf-8d37-515583d6b7a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-25381b71-5866f6986f-x77xg\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") " pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:19.681177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.681151    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls\") pod \"maas-api-e2e-mt-25381b71-97b79bbc5-2969x\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.779886 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.779839    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164de106-8245-4902-a61c-700304d5039c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.779886 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.779895    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr59b\" (UniqueName: \"kubernetes.io/projected/5b3c5409-70da-4caf-8d37-515583d6b7a5-kube-api-access-qr59b\") pod \"payload-processing-e2e-mt-25381b71-5866f6986f-x77xg\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") " pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:19.780161 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.780033    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twg28\" (UniqueName: \"kubernetes.io/projected/164de106-8245-4902-a61c-700304d5039c-kube-api-access-twg28\") pod \"payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.780161 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.780087    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3c5409-70da-4caf-8d37-515583d6b7a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-25381b71-5866f6986f-x77xg\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") " pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:19.780856 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.780831    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164de106-8245-4902-a61c-700304d5039c-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.780931 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.780869    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3c5409-70da-4caf-8d37-515583d6b7a5-plugins-config-volume\") pod \"payload-processing-e2e-mt-25381b71-5866f6986f-x77xg\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") " pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:19.790431 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.790404    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr59b\" (UniqueName: \"kubernetes.io/projected/5b3c5409-70da-4caf-8d37-515583d6b7a5-kube-api-access-qr59b\") pod \"payload-processing-e2e-mt-25381b71-5866f6986f-x77xg\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") " pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:19.790572 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.790430    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twg28\" (UniqueName: \"kubernetes.io/projected/164de106-8245-4902-a61c-700304d5039c-kube-api-access-twg28\") pod \"payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.861313 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.861210    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:19.895462 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.895400    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:19.926419 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:19.926105    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:20.138379 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:20.138356    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"]
Sep 10 15:49:20.139693 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:49:20.139652    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e8a0a86_b576_47d1_a9e3_4cf862287562.slice/crio-958997fa1f1423bdd8ff5867f32b86d18c39aebea7c8da0812a1d5eb30fb2e77 WatchSource:0}: Error finding container 958997fa1f1423bdd8ff5867f32b86d18c39aebea7c8da0812a1d5eb30fb2e77: Status 404 returned error can't find the container with id 958997fa1f1423bdd8ff5867f32b86d18c39aebea7c8da0812a1d5eb30fb2e77
Sep 10 15:49:20.189825 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:20.189798    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"]
Sep 10 15:49:20.190834 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:49:20.190809    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod164de106_8245_4902_a61c_700304d5039c.slice/crio-b11c88db40575cbc0e1e258e1dfeed6a6f55c0746ede4422c8541a6e4a614617 WatchSource:0}: Error finding container b11c88db40575cbc0e1e258e1dfeed6a6f55c0746ede4422c8541a6e4a614617: Status 404 returned error can't find the container with id b11c88db40575cbc0e1e258e1dfeed6a6f55c0746ede4422c8541a6e4a614617
Sep 10 15:49:20.214836 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:20.214814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"]
Sep 10 15:49:20.215840 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:49:20.215814    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b3c5409_70da_4caf_8d37_515583d6b7a5.slice/crio-7c1552cbf25d7fe200efaa7c4970c0753b68eb26909ca2f0328c620a494cbebe WatchSource:0}: Error finding container 7c1552cbf25d7fe200efaa7c4970c0753b68eb26909ca2f0328c620a494cbebe: Status 404 returned error can't find the container with id 7c1552cbf25d7fe200efaa7c4970c0753b68eb26909ca2f0328c620a494cbebe
Sep 10 15:49:21.082590 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.082551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x" event={"ID":"6e8a0a86-b576-47d1-a9e3-4cf862287562","Type":"ContainerStarted","Data":"1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c"}
Sep 10 15:49:21.083093 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.082613    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x" event={"ID":"6e8a0a86-b576-47d1-a9e3-4cf862287562","Type":"ContainerStarted","Data":"958997fa1f1423bdd8ff5867f32b86d18c39aebea7c8da0812a1d5eb30fb2e77"}
Sep 10 15:49:21.083093 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.082746    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:21.084297 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.084275    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz" event={"ID":"164de106-8245-4902-a61c-700304d5039c","Type":"ContainerStarted","Data":"bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba"}
Sep 10 15:49:21.084404 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.084322    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:21.084404 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.084342    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz" event={"ID":"164de106-8245-4902-a61c-700304d5039c","Type":"ContainerStarted","Data":"b11c88db40575cbc0e1e258e1dfeed6a6f55c0746ede4422c8541a6e4a614617"}
Sep 10 15:49:21.086056 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.086033    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg" event={"ID":"5b3c5409-70da-4caf-8d37-515583d6b7a5","Type":"ContainerStarted","Data":"a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551"}
Sep 10 15:49:21.086192 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.086072    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg" event={"ID":"5b3c5409-70da-4caf-8d37-515583d6b7a5","Type":"ContainerStarted","Data":"7c1552cbf25d7fe200efaa7c4970c0753b68eb26909ca2f0328c620a494cbebe"}
Sep 10 15:49:21.086192 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.086126    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:21.101530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.101485    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x" podStartSLOduration=2.7334015750000002 podStartE2EDuration="3.101471884s" totalImagesPullingTime="368.070309ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:21.099431353 +0000 UTC m=+1004.713558258" watchObservedRunningTime="2026-09-10 15:49:21.101471884 +0000 UTC m=+1004.715598787"
Sep 10 15:49:21.116803 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.116742    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg" podStartSLOduration=2.116729951 podStartE2EDuration="2.116729951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:21.115931116 +0000 UTC m=+1004.730058024" watchObservedRunningTime="2026-09-10 15:49:21.116729951 +0000 UTC m=+1004.730856855"
Sep 10 15:49:21.135372 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:21.135320    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz" podStartSLOduration=2.135302755 podStartE2EDuration="2.135302755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:21.134007634 +0000 UTC m=+1004.748134539" watchObservedRunningTime="2026-09-10 15:49:21.135302755 +0000 UTC m=+1004.749429661"
Sep 10 15:49:27.100999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:27.100970    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:49:32.096546 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:32.096511    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:49:32.097045 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:32.096686    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:49:54.194675 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:49:54.194637    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"]
Sep 10 15:50:00.916998 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:00.916961    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"]
Sep 10 15:50:00.917520 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:00.917229    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x" podUID="6e8a0a86-b576-47d1-a9e3-4cf862287562" containerName="maas-api" containerID="cri-o://1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c" gracePeriod=30
Sep 10 15:50:00.970323 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:00.970290    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"]
Sep 10 15:50:00.970595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:00.970548    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg" podUID="5b3c5409-70da-4caf-8d37-515583d6b7a5" containerName="payload-processing" containerID="cri-o://a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551" gracePeriod=30
Sep 10 15:50:00.978353 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:00.978255    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"]
Sep 10 15:50:00.978557 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:00.978497    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz" podUID="164de106-8245-4902-a61c-700304d5039c" containerName="payload-pre-processing" containerID="cri-o://bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba" gracePeriod=30
Sep 10 15:50:01.236136 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.236109    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:50:01.254863 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.254669    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls\") pod \"6e8a0a86-b576-47d1-a9e3-4cf862287562\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") "
Sep 10 15:50:01.255892 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.255122    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8a0a86-b576-47d1-a9e3-4cf862287562-openshift-service-ca\") pod \"6e8a0a86-b576-47d1-a9e3-4cf862287562\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") "
Sep 10 15:50:01.255892 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.255166    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9fb64\" (UniqueName: \"kubernetes.io/projected/6e8a0a86-b576-47d1-a9e3-4cf862287562-kube-api-access-9fb64\") pod \"6e8a0a86-b576-47d1-a9e3-4cf862287562\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") "
Sep 10 15:50:01.255892 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.255246    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-metrics-certs\") pod \"6e8a0a86-b576-47d1-a9e3-4cf862287562\" (UID: \"6e8a0a86-b576-47d1-a9e3-4cf862287562\") "
Sep 10 15:50:01.257403 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.257367    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e8a0a86-b576-47d1-a9e3-4cf862287562-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "6e8a0a86-b576-47d1-a9e3-4cf862287562" (UID: "6e8a0a86-b576-47d1-a9e3-4cf862287562"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:01.260028 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.259999    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "6e8a0a86-b576-47d1-a9e3-4cf862287562" (UID: "6e8a0a86-b576-47d1-a9e3-4cf862287562"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:01.261163 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.261136    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "6e8a0a86-b576-47d1-a9e3-4cf862287562" (UID: "6e8a0a86-b576-47d1-a9e3-4cf862287562"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:01.261396 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.261372    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8a0a86-b576-47d1-a9e3-4cf862287562-kube-api-access-9fb64" (OuterVolumeSpecName: "kube-api-access-9fb64") pod "6e8a0a86-b576-47d1-a9e3-4cf862287562" (UID: "6e8a0a86-b576-47d1-a9e3-4cf862287562"). InnerVolumeSpecName "kube-api-access-9fb64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:01.278051 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.278029    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:50:01.281402 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.281384    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:50:01.313988 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.313952    2577 generic.go:356] "Generic (PLEG): container finished" podID="5b3c5409-70da-4caf-8d37-515583d6b7a5" containerID="a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551" exitCode=0
Sep 10 15:50:01.314183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.314014    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"
Sep 10 15:50:01.314183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.314070    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg" event={"ID":"5b3c5409-70da-4caf-8d37-515583d6b7a5","Type":"ContainerDied","Data":"a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551"}
Sep 10 15:50:01.314183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.314160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg" event={"ID":"5b3c5409-70da-4caf-8d37-515583d6b7a5","Type":"ContainerDied","Data":"7c1552cbf25d7fe200efaa7c4970c0753b68eb26909ca2f0328c620a494cbebe"}
Sep 10 15:50:01.314359 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.314192    2577 scope.go:117] "RemoveContainer" containerID="a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551"
Sep 10 15:50:01.316182 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.316157    2577 generic.go:356] "Generic (PLEG): container finished" podID="6e8a0a86-b576-47d1-a9e3-4cf862287562" containerID="1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c" exitCode=0
Sep 10 15:50:01.316368 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.316278    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"
Sep 10 15:50:01.316914 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.316891    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x" event={"ID":"6e8a0a86-b576-47d1-a9e3-4cf862287562","Type":"ContainerDied","Data":"1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c"}
Sep 10 15:50:01.317033 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.316944    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x" event={"ID":"6e8a0a86-b576-47d1-a9e3-4cf862287562","Type":"ContainerDied","Data":"958997fa1f1423bdd8ff5867f32b86d18c39aebea7c8da0812a1d5eb30fb2e77"}
Sep 10 15:50:01.320521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.319553    2577 generic.go:356] "Generic (PLEG): container finished" podID="164de106-8245-4902-a61c-700304d5039c" containerID="bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba" exitCode=0
Sep 10 15:50:01.320521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.319590    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz" event={"ID":"164de106-8245-4902-a61c-700304d5039c","Type":"ContainerDied","Data":"bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba"}
Sep 10 15:50:01.320521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.319627    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz" event={"ID":"164de106-8245-4902-a61c-700304d5039c","Type":"ContainerDied","Data":"b11c88db40575cbc0e1e258e1dfeed6a6f55c0746ede4422c8541a6e4a614617"}
Sep 10 15:50:01.320521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.319693    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"
Sep 10 15:50:01.326371 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.326263    2577 scope.go:117] "RemoveContainer" containerID="a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551"
Sep 10 15:50:01.326583 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:01.326561    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551\": container with ID starting with a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551 not found: ID does not exist" containerID="a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551"
Sep 10 15:50:01.326632 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.326596    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551"} err="failed to get container status \"a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551\": rpc error: code = NotFound desc = could not find container \"a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551\": container with ID starting with a7a8c2d61ff29591b4f129006382da3801d481161221c94285bf31ca23679551 not found: ID does not exist"
Sep 10 15:50:01.326632 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.326619    2577 scope.go:117] "RemoveContainer" containerID="1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c"
Sep 10 15:50:01.335518 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.335498    2577 scope.go:117] "RemoveContainer" containerID="1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c"
Sep 10 15:50:01.335829 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:01.335810    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c\": container with ID starting with 1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c not found: ID does not exist" containerID="1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c"
Sep 10 15:50:01.335908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.335845    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c"} err="failed to get container status \"1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c\": rpc error: code = NotFound desc = could not find container \"1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c\": container with ID starting with 1299783b4d3343cb7e9e0a122184ea3f975005632dd2d30268c6fa8b649f115c not found: ID does not exist"
Sep 10 15:50:01.335908 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.335864    2577 scope.go:117] "RemoveContainer" containerID="bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba"
Sep 10 15:50:01.343888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.343871    2577 scope.go:117] "RemoveContainer" containerID="bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba"
Sep 10 15:50:01.344206 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:01.344189    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba\": container with ID starting with bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba not found: ID does not exist" containerID="bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba"
Sep 10 15:50:01.344265 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.344217    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba"} err="failed to get container status \"bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba\": rpc error: code = NotFound desc = could not find container \"bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba\": container with ID starting with bc62c63735893ff399cdb8b491982ba0757e24095f57f0a6e331edc210d959ba not found: ID does not exist"
Sep 10 15:50:01.350915 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.350893    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"]
Sep 10 15:50:01.356136 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356113    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-twg28\" (UniqueName: \"kubernetes.io/projected/164de106-8245-4902-a61c-700304d5039c-kube-api-access-twg28\") pod \"164de106-8245-4902-a61c-700304d5039c\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") "
Sep 10 15:50:01.356213 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356155    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qr59b\" (UniqueName: \"kubernetes.io/projected/5b3c5409-70da-4caf-8d37-515583d6b7a5-kube-api-access-qr59b\") pod \"5b3c5409-70da-4caf-8d37-515583d6b7a5\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") "
Sep 10 15:50:01.356250 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356213    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164de106-8245-4902-a61c-700304d5039c-plugins-config-volume\") pod \"164de106-8245-4902-a61c-700304d5039c\" (UID: \"164de106-8245-4902-a61c-700304d5039c\") "
Sep 10 15:50:01.356250 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356243    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3c5409-70da-4caf-8d37-515583d6b7a5-plugins-config-volume\") pod \"5b3c5409-70da-4caf-8d37-515583d6b7a5\" (UID: \"5b3c5409-70da-4caf-8d37-515583d6b7a5\") "
Sep 10 15:50:01.356348 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356286    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-25381b71-97b79bbc5-2969x"]
Sep 10 15:50:01.356536 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356520    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.356605 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356542    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6e8a0a86-b576-47d1-a9e3-4cf862287562-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.356605 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356574    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e8a0a86-b576-47d1-a9e3-4cf862287562-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.356605 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356583    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fb64\" (UniqueName: \"kubernetes.io/projected/6e8a0a86-b576-47d1-a9e3-4cf862287562-kube-api-access-9fb64\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.356715 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356597    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/164de106-8245-4902-a61c-700304d5039c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "164de106-8245-4902-a61c-700304d5039c" (UID: "164de106-8245-4902-a61c-700304d5039c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:01.356715 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.356639    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b3c5409-70da-4caf-8d37-515583d6b7a5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b3c5409-70da-4caf-8d37-515583d6b7a5" (UID: "5b3c5409-70da-4caf-8d37-515583d6b7a5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:01.358247 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.358229    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/164de106-8245-4902-a61c-700304d5039c-kube-api-access-twg28" (OuterVolumeSpecName: "kube-api-access-twg28") pod "164de106-8245-4902-a61c-700304d5039c" (UID: "164de106-8245-4902-a61c-700304d5039c"). InnerVolumeSpecName "kube-api-access-twg28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:01.358312 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.358294    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b3c5409-70da-4caf-8d37-515583d6b7a5-kube-api-access-qr59b" (OuterVolumeSpecName: "kube-api-access-qr59b") pod "5b3c5409-70da-4caf-8d37-515583d6b7a5" (UID: "5b3c5409-70da-4caf-8d37-515583d6b7a5"). InnerVolumeSpecName "kube-api-access-qr59b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:01.457850 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.457779    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twg28\" (UniqueName: \"kubernetes.io/projected/164de106-8245-4902-a61c-700304d5039c-kube-api-access-twg28\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.457850 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.457805    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr59b\" (UniqueName: \"kubernetes.io/projected/5b3c5409-70da-4caf-8d37-515583d6b7a5-kube-api-access-qr59b\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.457850 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.457814    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/164de106-8245-4902-a61c-700304d5039c-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.457850 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.457823    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b3c5409-70da-4caf-8d37-515583d6b7a5-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:01.635814 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.635788    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"]
Sep 10 15:50:01.639937 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.639912    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-25381b71-5866f6986f-x77xg"]
Sep 10 15:50:01.649494 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.649467    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"]
Sep 10 15:50:01.654205 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:01.654183    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-25381b71-6b87bcb64b-z26hz"]
Sep 10 15:50:02.908452 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:02.908420    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="164de106-8245-4902-a61c-700304d5039c" path="/var/lib/kubelet/pods/164de106-8245-4902-a61c-700304d5039c/volumes"
Sep 10 15:50:02.908848 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:02.908787    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b3c5409-70da-4caf-8d37-515583d6b7a5" path="/var/lib/kubelet/pods/5b3c5409-70da-4caf-8d37-515583d6b7a5/volumes"
Sep 10 15:50:02.909135 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:02.909122    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e8a0a86-b576-47d1-a9e3-4cf862287562" path="/var/lib/kubelet/pods/6e8a0a86-b576-47d1-a9e3-4cf862287562/volumes"
Sep 10 15:50:06.910609 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:06.910410    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"]
Sep 10 15:50:06.911123 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:06.910675    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt" podUID="bd823d89-2ba7-4753-bbd0-efb19a18d017" containerName="maas-api" containerID="cri-o://071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f" gracePeriod=30
Sep 10 15:50:06.964232 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:06.964196    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"]
Sep 10 15:50:06.964558 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:06.964508    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd" podUID="b57da81d-642d-4c11-976b-18a7da3f2790" containerName="payload-processing" containerID="cri-o://4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173" gracePeriod=30
Sep 10 15:50:06.969524 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:06.969494    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"]
Sep 10 15:50:06.969724 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:06.969697    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff" podUID="90afc743-ab68-4d63-a204-bc6d17e56966" containerName="payload-pre-processing" containerID="cri-o://62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25" gracePeriod=30
Sep 10 15:50:07.264316 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.264048    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:50:07.267936 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.267902    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:50:07.305976 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.305951    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rc4n2\" (UniqueName: \"kubernetes.io/projected/90afc743-ab68-4d63-a204-bc6d17e56966-kube-api-access-rc4n2\") pod \"90afc743-ab68-4d63-a204-bc6d17e56966\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") "
Sep 10 15:50:07.306159 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.306005    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90afc743-ab68-4d63-a204-bc6d17e56966-plugins-config-volume\") pod \"90afc743-ab68-4d63-a204-bc6d17e56966\" (UID: \"90afc743-ab68-4d63-a204-bc6d17e56966\") "
Sep 10 15:50:07.306159 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.306087    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57da81d-642d-4c11-976b-18a7da3f2790-plugins-config-volume\") pod \"b57da81d-642d-4c11-976b-18a7da3f2790\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") "
Sep 10 15:50:07.306159 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.306136    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pvmmq\" (UniqueName: \"kubernetes.io/projected/b57da81d-642d-4c11-976b-18a7da3f2790-kube-api-access-pvmmq\") pod \"b57da81d-642d-4c11-976b-18a7da3f2790\" (UID: \"b57da81d-642d-4c11-976b-18a7da3f2790\") "
Sep 10 15:50:07.306428 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.306400    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90afc743-ab68-4d63-a204-bc6d17e56966-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "90afc743-ab68-4d63-a204-bc6d17e56966" (UID: "90afc743-ab68-4d63-a204-bc6d17e56966"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:07.306572 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.306544    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57da81d-642d-4c11-976b-18a7da3f2790-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b57da81d-642d-4c11-976b-18a7da3f2790" (UID: "b57da81d-642d-4c11-976b-18a7da3f2790"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:07.308540 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.308503    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57da81d-642d-4c11-976b-18a7da3f2790-kube-api-access-pvmmq" (OuterVolumeSpecName: "kube-api-access-pvmmq") pod "b57da81d-642d-4c11-976b-18a7da3f2790" (UID: "b57da81d-642d-4c11-976b-18a7da3f2790"). InnerVolumeSpecName "kube-api-access-pvmmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:07.309065 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.309040    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90afc743-ab68-4d63-a204-bc6d17e56966-kube-api-access-rc4n2" (OuterVolumeSpecName: "kube-api-access-rc4n2") pod "90afc743-ab68-4d63-a204-bc6d17e56966" (UID: "90afc743-ab68-4d63-a204-bc6d17e56966"). InnerVolumeSpecName "kube-api-access-rc4n2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:07.349447 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.349418    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:50:07.360683 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.360659    2577 generic.go:356] "Generic (PLEG): container finished" podID="bd823d89-2ba7-4753-bbd0-efb19a18d017" containerID="071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f" exitCode=0
Sep 10 15:50:07.360817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.360701    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"
Sep 10 15:50:07.360817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.360715    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt" event={"ID":"bd823d89-2ba7-4753-bbd0-efb19a18d017","Type":"ContainerDied","Data":"071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f"}
Sep 10 15:50:07.360817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.360776    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt" event={"ID":"bd823d89-2ba7-4753-bbd0-efb19a18d017","Type":"ContainerDied","Data":"e92f8c11598a523d22a138a93e7620c4ea3164332ab8ce872a68126a35a1be0f"}
Sep 10 15:50:07.360817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.360804    2577 scope.go:117] "RemoveContainer" containerID="071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f"
Sep 10 15:50:07.362052 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.362025    2577 generic.go:356] "Generic (PLEG): container finished" podID="90afc743-ab68-4d63-a204-bc6d17e56966" containerID="62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25" exitCode=0
Sep 10 15:50:07.362142 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.362070    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"
Sep 10 15:50:07.362142 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.362113    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff" event={"ID":"90afc743-ab68-4d63-a204-bc6d17e56966","Type":"ContainerDied","Data":"62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25"}
Sep 10 15:50:07.362242 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.362160    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff" event={"ID":"90afc743-ab68-4d63-a204-bc6d17e56966","Type":"ContainerDied","Data":"6ac8252fc7209fcaecfbd045f1c41642e983b21921d59ef7a7c6e458b82e5d8f"}
Sep 10 15:50:07.363568 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.363540    2577 generic.go:356] "Generic (PLEG): container finished" podID="b57da81d-642d-4c11-976b-18a7da3f2790" containerID="4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173" exitCode=0
Sep 10 15:50:07.363686 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.363567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd" event={"ID":"b57da81d-642d-4c11-976b-18a7da3f2790","Type":"ContainerDied","Data":"4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173"}
Sep 10 15:50:07.363686 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.363591    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"
Sep 10 15:50:07.363686 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.363626    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd" event={"ID":"b57da81d-642d-4c11-976b-18a7da3f2790","Type":"ContainerDied","Data":"2b5b3a4d569ebaf05e7d6d58989fdb802bac3ad15adcd1ced214bdf9ea3e6050"}
Sep 10 15:50:07.372770 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.372727    2577 scope.go:117] "RemoveContainer" containerID="071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f"
Sep 10 15:50:07.373087 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:07.373062    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f\": container with ID starting with 071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f not found: ID does not exist" containerID="071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f"
Sep 10 15:50:07.373177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.373095    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f"} err="failed to get container status \"071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f\": rpc error: code = NotFound desc = could not find container \"071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f\": container with ID starting with 071deffd93edcd9621e2ac5daf8c93c2d068cead3dbe474d7aea8e7a1495775f not found: ID does not exist"
Sep 10 15:50:07.373177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.373125    2577 scope.go:117] "RemoveContainer" containerID="62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25"
Sep 10 15:50:07.382387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.382370    2577 scope.go:117] "RemoveContainer" containerID="62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25"
Sep 10 15:50:07.382659 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:07.382641    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25\": container with ID starting with 62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25 not found: ID does not exist" containerID="62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25"
Sep 10 15:50:07.382713 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.382665    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25"} err="failed to get container status \"62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25\": rpc error: code = NotFound desc = could not find container \"62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25\": container with ID starting with 62c9bd56a474812d660a59f12761237894644a8010792852b127ca0a1bf73c25 not found: ID does not exist"
Sep 10 15:50:07.382713 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.382680    2577 scope.go:117] "RemoveContainer" containerID="4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173"
Sep 10 15:50:07.388205 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.388171    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"]
Sep 10 15:50:07.394778 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.394737    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-80d47c-69c5bd6489-lfphd"]
Sep 10 15:50:07.394917 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.394898    2577 scope.go:117] "RemoveContainer" containerID="4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173"
Sep 10 15:50:07.395211 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:07.395188    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173\": container with ID starting with 4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173 not found: ID does not exist" containerID="4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173"
Sep 10 15:50:07.395295 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.395220    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173"} err="failed to get container status \"4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173\": rpc error: code = NotFound desc = could not find container \"4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173\": container with ID starting with 4721c6cbaee66a2b06f1e3ac0dd61e741e86a1d9ef111bf56a0a50ad9cd21173 not found: ID does not exist"
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.407390    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"]
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.407844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls\") pod \"bd823d89-2ba7-4753-bbd0-efb19a18d017\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") "
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.407917    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rg4vj\" (UniqueName: \"kubernetes.io/projected/bd823d89-2ba7-4753-bbd0-efb19a18d017-kube-api-access-rg4vj\") pod \"bd823d89-2ba7-4753-bbd0-efb19a18d017\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") "
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.408038    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-metrics-certs\") pod \"bd823d89-2ba7-4753-bbd0-efb19a18d017\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") "
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.408088    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd823d89-2ba7-4753-bbd0-efb19a18d017-openshift-service-ca\") pod \"bd823d89-2ba7-4753-bbd0-efb19a18d017\" (UID: \"bd823d89-2ba7-4753-bbd0-efb19a18d017\") "
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.408451    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57da81d-642d-4c11-976b-18a7da3f2790-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.408469    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvmmq\" (UniqueName: \"kubernetes.io/projected/b57da81d-642d-4c11-976b-18a7da3f2790-kube-api-access-pvmmq\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.408485    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rc4n2\" (UniqueName: \"kubernetes.io/projected/90afc743-ab68-4d63-a204-bc6d17e56966-kube-api-access-rc4n2\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.408500    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/90afc743-ab68-4d63-a204-bc6d17e56966-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.408863    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd823d89-2ba7-4753-bbd0-efb19a18d017-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bd823d89-2ba7-4753-bbd0-efb19a18d017" (UID: "bd823d89-2ba7-4753-bbd0-efb19a18d017"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:07.412993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.412990    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-80d47c-54bb695654-tpmff"]
Sep 10 15:50:07.414953 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.414927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd823d89-2ba7-4753-bbd0-efb19a18d017-kube-api-access-rg4vj" (OuterVolumeSpecName: "kube-api-access-rg4vj") pod "bd823d89-2ba7-4753-bbd0-efb19a18d017" (UID: "bd823d89-2ba7-4753-bbd0-efb19a18d017"). InnerVolumeSpecName "kube-api-access-rg4vj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:07.415032 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.414951    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bd823d89-2ba7-4753-bbd0-efb19a18d017" (UID: "bd823d89-2ba7-4753-bbd0-efb19a18d017"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:07.415100 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.415081    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bd823d89-2ba7-4753-bbd0-efb19a18d017" (UID: "bd823d89-2ba7-4753-bbd0-efb19a18d017"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:07.509795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.509692    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.509795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.509720    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd823d89-2ba7-4753-bbd0-efb19a18d017-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.509795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.509730    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bd823d89-2ba7-4753-bbd0-efb19a18d017-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.509795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.509738    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rg4vj\" (UniqueName: \"kubernetes.io/projected/bd823d89-2ba7-4753-bbd0-efb19a18d017-kube-api-access-rg4vj\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:07.681257 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.681227    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"]
Sep 10 15:50:07.685268 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:07.685245    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-80d47c-97bd48986-2wdtt"]
Sep 10 15:50:08.910451 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:08.910418    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90afc743-ab68-4d63-a204-bc6d17e56966" path="/var/lib/kubelet/pods/90afc743-ab68-4d63-a204-bc6d17e56966/volumes"
Sep 10 15:50:08.910980 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:08.910962    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b57da81d-642d-4c11-976b-18a7da3f2790" path="/var/lib/kubelet/pods/b57da81d-642d-4c11-976b-18a7da3f2790/volumes"
Sep 10 15:50:08.911453 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:08.911436    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd823d89-2ba7-4753-bbd0-efb19a18d017" path="/var/lib/kubelet/pods/bd823d89-2ba7-4753-bbd0-efb19a18d017/volumes"
Sep 10 15:50:09.216021 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.215925    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5" podUID="583e1afb-2da8-4c39-920d-3db301f63b55" containerName="main" containerID="cri-o://2ec7b353dd7afcff806ba28dedfb2fdd92978de3c61537876abce4d69b53f3f1" gracePeriod=45
Sep 10 15:50:09.379405 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.379372    2577 generic.go:356] "Generic (PLEG): container finished" podID="583e1afb-2da8-4c39-920d-3db301f63b55" containerID="2ec7b353dd7afcff806ba28dedfb2fdd92978de3c61537876abce4d69b53f3f1" exitCode=0
Sep 10 15:50:09.379543 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.379416    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5" event={"ID":"583e1afb-2da8-4c39-920d-3db301f63b55","Type":"ContainerDied","Data":"2ec7b353dd7afcff806ba28dedfb2fdd92978de3c61537876abce4d69b53f3f1"}
Sep 10 15:50:09.466304 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.466248    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:50:09.524181 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524154    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kbf2k\" (UniqueName: \"kubernetes.io/projected/583e1afb-2da8-4c39-920d-3db301f63b55-kube-api-access-kbf2k\") pod \"583e1afb-2da8-4c39-920d-3db301f63b55\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") "
Sep 10 15:50:09.524361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524201    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-tmp-dir\") pod \"583e1afb-2da8-4c39-920d-3db301f63b55\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") "
Sep 10 15:50:09.524361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524216    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-dshm\") pod \"583e1afb-2da8-4c39-920d-3db301f63b55\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") "
Sep 10 15:50:09.524361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524277    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e1afb-2da8-4c39-920d-3db301f63b55-tls-certs\") pod \"583e1afb-2da8-4c39-920d-3db301f63b55\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") "
Sep 10 15:50:09.524361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524300    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-model-cache\") pod \"583e1afb-2da8-4c39-920d-3db301f63b55\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") "
Sep 10 15:50:09.524361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524347    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-home\") pod \"583e1afb-2da8-4c39-920d-3db301f63b55\" (UID: \"583e1afb-2da8-4c39-920d-3db301f63b55\") "
Sep 10 15:50:09.524605 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524455    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "583e1afb-2da8-4c39-920d-3db301f63b55" (UID: "583e1afb-2da8-4c39-920d-3db301f63b55"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:09.524666 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524627    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-tmp-dir\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:09.524714 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524662    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-model-cache" (OuterVolumeSpecName: "model-cache") pod "583e1afb-2da8-4c39-920d-3db301f63b55" (UID: "583e1afb-2da8-4c39-920d-3db301f63b55"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:09.524714 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.524646    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-home" (OuterVolumeSpecName: "home") pod "583e1afb-2da8-4c39-920d-3db301f63b55" (UID: "583e1afb-2da8-4c39-920d-3db301f63b55"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:09.526663 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.526643    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/583e1afb-2da8-4c39-920d-3db301f63b55-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "583e1afb-2da8-4c39-920d-3db301f63b55" (UID: "583e1afb-2da8-4c39-920d-3db301f63b55"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:09.526780 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.526692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/583e1afb-2da8-4c39-920d-3db301f63b55-kube-api-access-kbf2k" (OuterVolumeSpecName: "kube-api-access-kbf2k") pod "583e1afb-2da8-4c39-920d-3db301f63b55" (UID: "583e1afb-2da8-4c39-920d-3db301f63b55"). InnerVolumeSpecName "kube-api-access-kbf2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:09.526780 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.526720    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-dshm" (OuterVolumeSpecName: "dshm") pod "583e1afb-2da8-4c39-920d-3db301f63b55" (UID: "583e1afb-2da8-4c39-920d-3db301f63b55"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:50:09.625437 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.625400    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-model-cache\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:09.625437 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.625429    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-home\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:09.625437 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.625438    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbf2k\" (UniqueName: \"kubernetes.io/projected/583e1afb-2da8-4c39-920d-3db301f63b55-kube-api-access-kbf2k\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:09.625437 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.625446    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/583e1afb-2da8-4c39-920d-3db301f63b55-dshm\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:09.625698 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:09.625456    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/583e1afb-2da8-4c39-920d-3db301f63b55-tls-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:10.386204 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:10.386172    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"
Sep 10 15:50:10.386679 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:10.386206    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5" event={"ID":"583e1afb-2da8-4c39-920d-3db301f63b55","Type":"ContainerDied","Data":"16a09f5cd6ee769de601f779e6d5b583921f258d9f07685cff5003f9372071cf"}
Sep 10 15:50:10.386679 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:10.386265    2577 scope.go:117] "RemoveContainer" containerID="2ec7b353dd7afcff806ba28dedfb2fdd92978de3c61537876abce4d69b53f3f1"
Sep 10 15:50:10.407740 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:10.407713    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"]
Sep 10 15:50:10.409705 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:10.409684    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-80d47c/test-model-80d47c-kserve-748496b457-4vxn5"]
Sep 10 15:50:10.908795 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:10.908739    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="583e1afb-2da8-4c39-920d-3db301f63b55" path="/var/lib/kubelet/pods/583e1afb-2da8-4c39-920d-3db301f63b55/volumes"
Sep 10 15:50:24.323201 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.323168    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"]
Sep 10 15:50:24.323586 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.323404    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p" podUID="81cc378d-f314-45d7-aced-e3e742d75c39" containerName="payload-pre-processing" containerID="cri-o://befdc7341542dcf83b653ca22416b9105cc4111c4c02e3d03c831fbe5eb01d81" gracePeriod=30
Sep 10 15:50:24.471699 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.471664    2577 generic.go:356] "Generic (PLEG): container finished" podID="81cc378d-f314-45d7-aced-e3e742d75c39" containerID="befdc7341542dcf83b653ca22416b9105cc4111c4c02e3d03c831fbe5eb01d81" exitCode=0
Sep 10 15:50:24.471889 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.471733    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p" event={"ID":"81cc378d-f314-45d7-aced-e3e742d75c39","Type":"ContainerDied","Data":"befdc7341542dcf83b653ca22416b9105cc4111c4c02e3d03c831fbe5eb01d81"}
Sep 10 15:50:24.584116 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.584094    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:50:24.646029 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.645992    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81cc378d-f314-45d7-aced-e3e742d75c39-plugins-config-volume\") pod \"81cc378d-f314-45d7-aced-e3e742d75c39\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") "
Sep 10 15:50:24.646200 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.646094    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47fjb\" (UniqueName: \"kubernetes.io/projected/81cc378d-f314-45d7-aced-e3e742d75c39-kube-api-access-47fjb\") pod \"81cc378d-f314-45d7-aced-e3e742d75c39\" (UID: \"81cc378d-f314-45d7-aced-e3e742d75c39\") "
Sep 10 15:50:24.646356 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.646331    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cc378d-f314-45d7-aced-e3e742d75c39-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81cc378d-f314-45d7-aced-e3e742d75c39" (UID: "81cc378d-f314-45d7-aced-e3e742d75c39"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:50:24.648344 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.648320    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cc378d-f314-45d7-aced-e3e742d75c39-kube-api-access-47fjb" (OuterVolumeSpecName: "kube-api-access-47fjb") pod "81cc378d-f314-45d7-aced-e3e742d75c39" (UID: "81cc378d-f314-45d7-aced-e3e742d75c39"). InnerVolumeSpecName "kube-api-access-47fjb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:24.746574 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.746540    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47fjb\" (UniqueName: \"kubernetes.io/projected/81cc378d-f314-45d7-aced-e3e742d75c39-kube-api-access-47fjb\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:24.746574 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:24.746567    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81cc378d-f314-45d7-aced-e3e742d75c39-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:50:25.478933 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:25.478899    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"
Sep 10 15:50:25.479399 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:25.478936    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p" event={"ID":"81cc378d-f314-45d7-aced-e3e742d75c39","Type":"ContainerDied","Data":"eb8cf80bb8170d29444f278f8386d663273658ff59f6cb36c50945d4b8b99e4d"}
Sep 10 15:50:25.479399 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:25.479004    2577 scope.go:117] "RemoveContainer" containerID="befdc7341542dcf83b653ca22416b9105cc4111c4c02e3d03c831fbe5eb01d81"
Sep 10 15:50:25.497520 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:25.497496    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"]
Sep 10 15:50:25.501428 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:25.501401    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-8a850d-55887f7c54-nhf6p"]
Sep 10 15:50:26.367379 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367344    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"]
Sep 10 15:50:26.367926 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367904    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b3c5409-70da-4caf-8d37-515583d6b7a5" containerName="payload-processing"
Sep 10 15:50:26.367996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367929    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b3c5409-70da-4caf-8d37-515583d6b7a5" containerName="payload-processing"
Sep 10 15:50:26.367996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367950    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81cc378d-f314-45d7-aced-e3e742d75c39" containerName="payload-pre-processing"
Sep 10 15:50:26.367996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367958    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="81cc378d-f314-45d7-aced-e3e742d75c39" containerName="payload-pre-processing"
Sep 10 15:50:26.367996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367969    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="583e1afb-2da8-4c39-920d-3db301f63b55" containerName="main"
Sep 10 15:50:26.367996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367977    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="583e1afb-2da8-4c39-920d-3db301f63b55" containerName="main"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.367992    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b57da81d-642d-4c11-976b-18a7da3f2790" containerName="payload-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368007    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b57da81d-642d-4c11-976b-18a7da3f2790" containerName="payload-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368017    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd823d89-2ba7-4753-bbd0-efb19a18d017" containerName="maas-api"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368025    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd823d89-2ba7-4753-bbd0-efb19a18d017" containerName="maas-api"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368042    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90afc743-ab68-4d63-a204-bc6d17e56966" containerName="payload-pre-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368050    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="90afc743-ab68-4d63-a204-bc6d17e56966" containerName="payload-pre-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368060    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="164de106-8245-4902-a61c-700304d5039c" containerName="payload-pre-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368068    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="164de106-8245-4902-a61c-700304d5039c" containerName="payload-pre-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368084    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8a0a86-b576-47d1-a9e3-4cf862287562" containerName="maas-api"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368092    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8a0a86-b576-47d1-a9e3-4cf862287562" containerName="maas-api"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368210    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd823d89-2ba7-4753-bbd0-efb19a18d017" containerName="maas-api"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368221    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8a0a86-b576-47d1-a9e3-4cf862287562" containerName="maas-api"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368233    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="90afc743-ab68-4d63-a204-bc6d17e56966" containerName="payload-pre-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368245    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b3c5409-70da-4caf-8d37-515583d6b7a5" containerName="payload-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368255    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b57da81d-642d-4c11-976b-18a7da3f2790" containerName="payload-processing"
Sep 10 15:50:26.368262 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368263    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="81cc378d-f314-45d7-aced-e3e742d75c39" containerName="payload-pre-processing"
Sep 10 15:50:26.369030 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368272    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="164de106-8245-4902-a61c-700304d5039c" containerName="payload-pre-processing"
Sep 10 15:50:26.369030 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.368281    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="583e1afb-2da8-4c39-920d-3db301f63b55" containerName="main"
Sep 10 15:50:26.379712 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.379683    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.380185 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.379692    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"]
Sep 10 15:50:26.384304 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.384275    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w2-8e3dc2-serving-cert\""
Sep 10 15:50:26.461556 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.461520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da59559-f754-4b63-bd70-b0c0fe83c6a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.461728 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.461605    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.461728 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.461636    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dws2g\" (UniqueName: \"kubernetes.io/projected/8da59559-f754-4b63-bd70-b0c0fe83c6a5-kube-api-access-dws2g\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.461728 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.461658    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.562920 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.562884    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.562920 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.562923    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dws2g\" (UniqueName: \"kubernetes.io/projected/8da59559-f754-4b63-bd70-b0c0fe83c6a5-kube-api-access-dws2g\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.563421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.562941    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.563421 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.562983    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da59559-f754-4b63-bd70-b0c0fe83c6a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.563421 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:26.563055    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-serving-cert: secret "maas-api-e2e-shared-a-w2-8e3dc2-serving-cert" not found
Sep 10 15:50:26.563421 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:26.563134    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls podName:8da59559-f754-4b63-bd70-b0c0fe83c6a5 nodeName:}" failed. No retries permitted until 2026-09-10 15:50:27.063117123 +0000 UTC m=+1070.677244005 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls") pod "maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb" (UID: "8da59559-f754-4b63-bd70-b0c0fe83c6a5") : secret "maas-api-e2e-shared-a-w2-8e3dc2-serving-cert" not found
Sep 10 15:50:26.563607 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.563589    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da59559-f754-4b63-bd70-b0c0fe83c6a5-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.565607 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.565579    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-metrics-certs\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.571873 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.571841    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dws2g\" (UniqueName: \"kubernetes.io/projected/8da59559-f754-4b63-bd70-b0c0fe83c6a5-kube-api-access-dws2g\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:26.908610 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:26.908581    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81cc378d-f314-45d7-aced-e3e742d75c39" path="/var/lib/kubelet/pods/81cc378d-f314-45d7-aced-e3e742d75c39/volumes"
Sep 10 15:50:27.020088 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.020053    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"]
Sep 10 15:50:27.024432 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.024415    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.026875 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.026852    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-8e3dc2\""
Sep 10 15:50:27.027014 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.026955    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-8e3dc2-dockercfg-vwp5f\""
Sep 10 15:50:27.031796 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.031771    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"]
Sep 10 15:50:27.066979 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.066938    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nx6fs\" (UniqueName: \"kubernetes.io/projected/acbfcb1b-4c71-4575-a4df-976e4de543c8-kube-api-access-nx6fs\") pod \"payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.067116 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.067049    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:27.067116 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.067081    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbfcb1b-4c71-4575-a4df-976e4de543c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.069522 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.069495    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls\") pod \"maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:27.167530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.167443    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nx6fs\" (UniqueName: \"kubernetes.io/projected/acbfcb1b-4c71-4575-a4df-976e4de543c8-kube-api-access-nx6fs\") pod \"payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.167698 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.167528    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbfcb1b-4c71-4575-a4df-976e4de543c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.168220 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.168196    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbfcb1b-4c71-4575-a4df-976e4de543c8-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.175825 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.175804    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nx6fs\" (UniqueName: \"kubernetes.io/projected/acbfcb1b-4c71-4575-a4df-976e4de543c8-kube-api-access-nx6fs\") pod \"payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.295329 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.295297    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:27.335333 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.335296    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:27.573286 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.573239    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"]
Sep 10 15:50:27.574473 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:50:27.574452    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8da59559_f754_4b63_bd70_b0c0fe83c6a5.slice/crio-ff22b7333bf5e846abd6e0e3dad5c9671589cb41046849daa1523b6819ac3484 WatchSource:0}: Error finding container ff22b7333bf5e846abd6e0e3dad5c9671589cb41046849daa1523b6819ac3484: Status 404 returned error can't find the container with id ff22b7333bf5e846abd6e0e3dad5c9671589cb41046849daa1523b6819ac3484
Sep 10 15:50:27.575801 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.575784    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:50:27.619558 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:27.619530    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"]
Sep 10 15:50:27.620909 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:50:27.620876    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacbfcb1b_4c71_4575_a4df_976e4de543c8.slice/crio-16f761815872038784c01b6d798873ca9687e7df1b844aff7f474e6de64f651c WatchSource:0}: Error finding container 16f761815872038784c01b6d798873ca9687e7df1b844aff7f474e6de64f651c: Status 404 returned error can't find the container with id 16f761815872038784c01b6d798873ca9687e7df1b844aff7f474e6de64f651c
Sep 10 15:50:28.499182 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.499085    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt" event={"ID":"acbfcb1b-4c71-4575-a4df-976e4de543c8","Type":"ContainerStarted","Data":"2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0"}
Sep 10 15:50:28.499182 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.499151    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:28.499182 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.499172    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt" event={"ID":"acbfcb1b-4c71-4575-a4df-976e4de543c8","Type":"ContainerStarted","Data":"16f761815872038784c01b6d798873ca9687e7df1b844aff7f474e6de64f651c"}
Sep 10 15:50:28.500982 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.500957    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb" event={"ID":"8da59559-f754-4b63-bd70-b0c0fe83c6a5","Type":"ContainerStarted","Data":"7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be"}
Sep 10 15:50:28.501124 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.501002    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb" event={"ID":"8da59559-f754-4b63-bd70-b0c0fe83c6a5","Type":"ContainerStarted","Data":"ff22b7333bf5e846abd6e0e3dad5c9671589cb41046849daa1523b6819ac3484"}
Sep 10 15:50:28.501190 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.501122    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:28.517172 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.517093    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt" podStartSLOduration=1.517074188 podStartE2EDuration="1.517074188s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:28.513709054 +0000 UTC m=+1072.127835958" watchObservedRunningTime="2026-09-10 15:50:28.517074188 +0000 UTC m=+1072.131201089"
Sep 10 15:50:28.530816 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:28.530766    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb" podStartSLOduration=1.899246107 podStartE2EDuration="2.530739319s" totalImagesPullingTime="631.493212ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:28.528576194 +0000 UTC m=+1072.142703099" watchObservedRunningTime="2026-09-10 15:50:28.530739319 +0000 UTC m=+1072.144866252"
Sep 10 15:50:34.514114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:34.514086    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:50:36.370727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.370688    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"]
Sep 10 15:50:36.373641 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.373621    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.376165 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.376143    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-6f8c71-serving-cert\""
Sep 10 15:50:36.384278 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.384210    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"]
Sep 10 15:50:36.449790 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.449728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.449952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.449832    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd7ls\" (UniqueName: \"kubernetes.io/projected/bdeb1ef0-1051-4025-8ba3-468720297c54-kube-api-access-sd7ls\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.449952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.449863    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeb1ef0-1051-4025-8ba3-468720297c54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.449952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.449934    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.550583 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.550553    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.550779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.550590    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd7ls\" (UniqueName: \"kubernetes.io/projected/bdeb1ef0-1051-4025-8ba3-468720297c54-kube-api-access-sd7ls\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.550779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.550612    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeb1ef0-1051-4025-8ba3-468720297c54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.550779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.550650    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.550779 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:36.550711    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-serving-cert: secret "maas-api-e2e-shared-a-w3-6f8c71-serving-cert" not found
Sep 10 15:50:36.550960 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:36.550812    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls podName:bdeb1ef0-1051-4025-8ba3-468720297c54 nodeName:}" failed. No retries permitted until 2026-09-10 15:50:37.050788694 +0000 UTC m=+1080.664915589 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls") pod "maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq" (UID: "bdeb1ef0-1051-4025-8ba3-468720297c54") : secret "maas-api-e2e-shared-a-w3-6f8c71-serving-cert" not found
Sep 10 15:50:36.551408 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.551384    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeb1ef0-1051-4025-8ba3-468720297c54-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.553207 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.553186    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:36.558613 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:36.558580    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd7ls\" (UniqueName: \"kubernetes.io/projected/bdeb1ef0-1051-4025-8ba3-468720297c54-kube-api-access-sd7ls\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:37.003321 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.003286    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"]
Sep 10 15:50:37.005972 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.005950    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.008346 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.008322    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-6f8c71\""
Sep 10 15:50:37.008434 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.008410    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-6f8c71-dockercfg-gt8bp\""
Sep 10 15:50:37.017278 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.016816    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"]
Sep 10 15:50:37.054437 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.054412    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:37.054530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.054471    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cr7k\" (UniqueName: \"kubernetes.io/projected/9aee3627-d9b2-4307-be76-d3f1e73d3130-kube-api-access-8cr7k\") pod \"payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.054530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.054520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aee3627-d9b2-4307-be76-d3f1e73d3130-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.056956 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.056927    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:37.155779 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.155710    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aee3627-d9b2-4307-be76-d3f1e73d3130-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.155975 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.155868    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cr7k\" (UniqueName: \"kubernetes.io/projected/9aee3627-d9b2-4307-be76-d3f1e73d3130-kube-api-access-8cr7k\") pod \"payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.156376 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.156355    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aee3627-d9b2-4307-be76-d3f1e73d3130-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.167857 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.167833    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cr7k\" (UniqueName: \"kubernetes.io/projected/9aee3627-d9b2-4307-be76-d3f1e73d3130-kube-api-access-8cr7k\") pod \"payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.288578 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.288472    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:37.318160 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.318132    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:37.563140 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.563116    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"]
Sep 10 15:50:37.565335 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:50:37.565311    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdeb1ef0_1051_4025_8ba3_468720297c54.slice/crio-cb03d73be26bb8fc151c596060cb8dd7eff378326026a3cb3e383e924ceb6918 WatchSource:0}: Error finding container cb03d73be26bb8fc151c596060cb8dd7eff378326026a3cb3e383e924ceb6918: Status 404 returned error can't find the container with id cb03d73be26bb8fc151c596060cb8dd7eff378326026a3cb3e383e924ceb6918
Sep 10 15:50:37.597545 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:37.597518    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"]
Sep 10 15:50:37.598193 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:50:37.598171    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9aee3627_d9b2_4307_be76_d3f1e73d3130.slice/crio-fed6d4d978747cb4d4fd9af67b8f99b7a6873f772b06dbcc249670cc5aab9585 WatchSource:0}: Error finding container fed6d4d978747cb4d4fd9af67b8f99b7a6873f772b06dbcc249670cc5aab9585: Status 404 returned error can't find the container with id fed6d4d978747cb4d4fd9af67b8f99b7a6873f772b06dbcc249670cc5aab9585
Sep 10 15:50:38.558825 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.558785    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq" event={"ID":"bdeb1ef0-1051-4025-8ba3-468720297c54","Type":"ContainerStarted","Data":"4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc"}
Sep 10 15:50:38.559022 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.558857    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:38.559022 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.558880    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq" event={"ID":"bdeb1ef0-1051-4025-8ba3-468720297c54","Type":"ContainerStarted","Data":"cb03d73be26bb8fc151c596060cb8dd7eff378326026a3cb3e383e924ceb6918"}
Sep 10 15:50:38.560326 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.560304    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9" event={"ID":"9aee3627-d9b2-4307-be76-d3f1e73d3130","Type":"ContainerStarted","Data":"2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882"}
Sep 10 15:50:38.560440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.560343    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9" event={"ID":"9aee3627-d9b2-4307-be76-d3f1e73d3130","Type":"ContainerStarted","Data":"fed6d4d978747cb4d4fd9af67b8f99b7a6873f772b06dbcc249670cc5aab9585"}
Sep 10 15:50:38.560440 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.560432    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:38.597184 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.597125    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq" podStartSLOduration=1.7945577080000001 podStartE2EDuration="2.597109608s" totalImagesPullingTime="802.5519ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:38.594312864 +0000 UTC m=+1082.208439768" watchObservedRunningTime="2026-09-10 15:50:38.597109608 +0000 UTC m=+1082.211236509"
Sep 10 15:50:38.620355 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:38.620233    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9" podStartSLOduration=2.620214197 podStartE2EDuration="2.620214197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:38.619953721 +0000 UTC m=+1082.234080627" watchObservedRunningTime="2026-09-10 15:50:38.620214197 +0000 UTC m=+1082.234341106"
Sep 10 15:50:39.510547 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:39.510515    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:50:44.571187 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:44.571154    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 15:50:49.568522 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:49.568492    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 15:50:54.468271 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.468239    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"]
Sep 10 15:50:54.471828 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.471805    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.474273 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.474245    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-761990-serving-cert\""
Sep 10 15:50:54.482720 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.482693    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"]
Sep 10 15:50:54.602859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.602820    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.602859 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.602859    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.603113 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.602926    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7p4hp\" (UniqueName: \"kubernetes.io/projected/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-kube-api-access-7p4hp\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.603113 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.602973    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.703993 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.703950    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.704214 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.704004    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.704214 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.704062    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7p4hp\" (UniqueName: \"kubernetes.io/projected/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-kube-api-access-7p4hp\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.704214 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.704116    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.704381 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:54.704258    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-serving-cert: secret "maas-api-e2e-shared-b-w2-761990-serving-cert" not found
Sep 10 15:50:54.704381 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:50:54.704333    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls podName:7e2307c6-a64d-406a-a3b5-e6ce7992e96f nodeName:}" failed. No retries permitted until 2026-09-10 15:50:55.204311369 +0000 UTC m=+1098.818438257 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls") pod "maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt" (UID: "7e2307c6-a64d-406a-a3b5-e6ce7992e96f") : secret "maas-api-e2e-shared-b-w2-761990-serving-cert" not found
Sep 10 15:50:54.704704 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.704680    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.706798 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.706775    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:54.713266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:54.713246    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7p4hp\" (UniqueName: \"kubernetes.io/projected/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-kube-api-access-7p4hp\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:55.208802 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:55.208765    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:55.211209 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:55.211176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:55.385274 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:55.385233    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:55.659241 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:55.659220    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"]
Sep 10 15:50:55.660623 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:50:55.660602    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e2307c6_a64d_406a_a3b5_e6ce7992e96f.slice/crio-a6914ad0ffee6c31c26522c79effcf36d677031b3245208eccef9eadb93a4e3d WatchSource:0}: Error finding container a6914ad0ffee6c31c26522c79effcf36d677031b3245208eccef9eadb93a4e3d: Status 404 returned error can't find the container with id a6914ad0ffee6c31c26522c79effcf36d677031b3245208eccef9eadb93a4e3d
Sep 10 15:50:56.665369 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:56.665337    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt" event={"ID":"7e2307c6-a64d-406a-a3b5-e6ce7992e96f","Type":"ContainerStarted","Data":"00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470"}
Sep 10 15:50:56.665788 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:56.665392    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt" event={"ID":"7e2307c6-a64d-406a-a3b5-e6ce7992e96f","Type":"ContainerStarted","Data":"a6914ad0ffee6c31c26522c79effcf36d677031b3245208eccef9eadb93a4e3d"}
Sep 10 15:50:56.665788 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:56.665549    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:50:56.684016 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:50:56.683963    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt" podStartSLOduration=2.384573521 podStartE2EDuration="2.683943871s" totalImagesPullingTime="299.37035ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:56.680997123 +0000 UTC m=+1100.295124029" watchObservedRunningTime="2026-09-10 15:50:56.683943871 +0000 UTC m=+1100.298070775"
Sep 10 15:51:02.675984 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:02.675937    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:51:04.453171 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.453134    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"]
Sep 10 15:51:04.457216 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.457195    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.460639 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.460608    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-e412c4-serving-cert\""
Sep 10 15:51:04.469807 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.469780    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"]
Sep 10 15:51:04.594032 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.593987    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.594032 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.594032    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.594266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.594152    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drxvp\" (UniqueName: \"kubernetes.io/projected/13c276c6-b209-4f24-84e4-ccb1414ed004-kube-api-access-drxvp\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.594266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.594219    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c276c6-b209-4f24-84e4-ccb1414ed004-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.694971 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.694926    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.694971 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.694976    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.695238 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.695037    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drxvp\" (UniqueName: \"kubernetes.io/projected/13c276c6-b209-4f24-84e4-ccb1414ed004-kube-api-access-drxvp\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.695238 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.695088    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c276c6-b209-4f24-84e4-ccb1414ed004-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.695238 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:51:04.695094    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-serving-cert: secret "maas-api-e2e-shared-b-w3-e412c4-serving-cert" not found
Sep 10 15:51:04.695238 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:51:04.695171    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls podName:13c276c6-b209-4f24-84e4-ccb1414ed004 nodeName:}" failed. No retries permitted until 2026-09-10 15:51:05.195149452 +0000 UTC m=+1108.809276342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls") pod "maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk" (UID: "13c276c6-b209-4f24-84e4-ccb1414ed004") : secret "maas-api-e2e-shared-b-w3-e412c4-serving-cert" not found
Sep 10 15:51:04.695844 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.695822    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c276c6-b209-4f24-84e4-ccb1414ed004-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.697928 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.697897    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:04.709501 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:04.709444    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drxvp\" (UniqueName: \"kubernetes.io/projected/13c276c6-b209-4f24-84e4-ccb1414ed004-kube-api-access-drxvp\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:05.201037 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:05.200986    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:05.203603 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:05.203570    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:05.369379 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:05.369341    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:05.643359 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:05.643333    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"]
Sep 10 15:51:05.644625 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:51:05.644600    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13c276c6_b209_4f24_84e4_ccb1414ed004.slice/crio-918cb4744c79d19060600ddc996d00ad93d9dfc9560933c9d66ebb27439f0c9e WatchSource:0}: Error finding container 918cb4744c79d19060600ddc996d00ad93d9dfc9560933c9d66ebb27439f0c9e: Status 404 returned error can't find the container with id 918cb4744c79d19060600ddc996d00ad93d9dfc9560933c9d66ebb27439f0c9e
Sep 10 15:51:05.717146 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:05.717116    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk" event={"ID":"13c276c6-b209-4f24-84e4-ccb1414ed004","Type":"ContainerStarted","Data":"918cb4744c79d19060600ddc996d00ad93d9dfc9560933c9d66ebb27439f0c9e"}
Sep 10 15:51:06.724220 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:06.724182    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk" event={"ID":"13c276c6-b209-4f24-84e4-ccb1414ed004","Type":"ContainerStarted","Data":"325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694"}
Sep 10 15:51:06.724646 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:06.724293    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:06.741307 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:06.741258    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk" podStartSLOduration=2.419321026 podStartE2EDuration="2.741243938s" totalImagesPullingTime="321.922912ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:06.739516008 +0000 UTC m=+1110.353642924" watchObservedRunningTime="2026-09-10 15:51:06.741243938 +0000 UTC m=+1110.355370841"
Sep 10 15:51:12.734585 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:12.734552    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 15:51:39.949448 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:39.949412    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"]
Sep 10 15:51:39.953818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:39.953801    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:39.957105 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:39.957068    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"auth-test-model-6f8c71-kserve-self-signed-certs\""
Sep 10 15:51:39.957105 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:39.957082    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"kube-root-ca.crt\""
Sep 10 15:51:39.957300 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:39.957254    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"openshift-service-ca.crt\""
Sep 10 15:51:39.957361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:39.957322    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"default-dockercfg-g7czv\""
Sep 10 15:51:39.960207 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:39.960186    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"]
Sep 10 15:51:40.104437 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.104403    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kc5kj\" (UniqueName: \"kubernetes.io/projected/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-kube-api-access-kc5kj\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.104609 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.104529    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-dshm\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.104609 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.104554    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-model-cache\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.104609 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.104580    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tmp-dir\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.104609 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.104601    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tls-certs\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.104834 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.104621    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-home\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205334 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205248    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-dshm\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205334 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-model-cache\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205334 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205303    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tmp-dir\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205334 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205330    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tls-certs\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205640 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205358    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-home\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205640 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205394    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kc5kj\" (UniqueName: \"kubernetes.io/projected/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-kube-api-access-kc5kj\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205776 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205739    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-model-cache\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205795    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tmp-dir\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.205899 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.205865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-home\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.207639 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.207616    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-dshm\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.207891 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.207874    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tls-certs\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.213288 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.213265    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kc5kj\" (UniqueName: \"kubernetes.io/projected/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-kube-api-access-kc5kj\") pod \"auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.266370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.266347    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:40.539027 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.539005    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"]
Sep 10 15:51:40.540594 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:51:40.540573    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2f6c680e_eced_4a1c_9a33_a9e71c9da2e3.slice/crio-c457ae42b1ce1bb61a56386a8a5206804e9ac1373590ee170a52350cf145ac98 WatchSource:0}: Error finding container c457ae42b1ce1bb61a56386a8a5206804e9ac1373590ee170a52350cf145ac98: Status 404 returned error can't find the container with id c457ae42b1ce1bb61a56386a8a5206804e9ac1373590ee170a52350cf145ac98
Sep 10 15:51:40.913793 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.913741    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9" event={"ID":"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3","Type":"ContainerStarted","Data":"b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74"}
Sep 10 15:51:40.913945 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.913804    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9" event={"ID":"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3","Type":"ContainerStarted","Data":"c457ae42b1ce1bb61a56386a8a5206804e9ac1373590ee170a52350cf145ac98"}
Sep 10 15:51:40.930495 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:40.930445    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9" podStartSLOduration=1.930428729 podStartE2EDuration="1.930428729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:40.928230758 +0000 UTC m=+1144.542357660" watchObservedRunningTime="2026-09-10 15:51:40.930428729 +0000 UTC m=+1144.544555632"
Sep 10 15:51:45.872226 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:45.872194    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"]
Sep 10 15:51:45.876251 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:45.876225    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:45.878944 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:45.878923    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-c1384d3d-serving-cert\""
Sep 10 15:51:45.885093 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:45.885032    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"]
Sep 10 15:51:46.063103 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.063054    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-metrics-certs\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.063293 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.063234    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-maas-api-tls\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.063383 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.063351    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-openshift-service-ca\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.063609 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.063556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt5jz\" (UniqueName: \"kubernetes.io/projected/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-kube-api-access-kt5jz\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.164813 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.164721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-maas-api-tls\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.164813 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.164781    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-openshift-service-ca\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.165017 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.164879    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt5jz\" (UniqueName: \"kubernetes.io/projected/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-kube-api-access-kt5jz\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.165017 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.164905    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-metrics-certs\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.165578 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.165554    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-openshift-service-ca\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.167258 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.167231    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-metrics-certs\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.167361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.167266    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-maas-api-tls\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.176703 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.176667    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt5jz\" (UniqueName: \"kubernetes.io/projected/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-kube-api-access-kt5jz\") pod \"maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.191569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.191545    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:46.469783 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.469730    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"]
Sep 10 15:51:46.471032 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:51:46.471009    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2428ae1_bfd1_41ff_9f6f_df912cfee9b4.slice/crio-60092af08eada0115123d437bf2158588f454a7a879b982b374ea01633daafa7 WatchSource:0}: Error finding container 60092af08eada0115123d437bf2158588f454a7a879b982b374ea01633daafa7: Status 404 returned error can't find the container with id 60092af08eada0115123d437bf2158588f454a7a879b982b374ea01633daafa7
Sep 10 15:51:46.947065 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:46.947030    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh" event={"ID":"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4","Type":"ContainerStarted","Data":"60092af08eada0115123d437bf2158588f454a7a879b982b374ea01633daafa7"}
Sep 10 15:51:47.953832 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:47.953796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh" event={"ID":"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4","Type":"ContainerStarted","Data":"a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d"}
Sep 10 15:51:47.954205 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:47.953976    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:51:47.972029 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:47.971982    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh" podStartSLOduration=2.411362512 podStartE2EDuration="2.971968622s" totalImagesPullingTime="560.60611ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:51:47.969364475 +0000 UTC m=+1151.583491381" watchObservedRunningTime="2026-09-10 15:51:47.971968622 +0000 UTC m=+1151.586095526"
Sep 10 15:51:50.267105 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:50.267069    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:50.267509 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:50.267123    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:50.279644 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:50.279617    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:50.980701 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:50.980676    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:51:53.965899 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:51:53.965868    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:53:04.699981 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:04.699950    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"]
Sep 10 15:53:04.700578 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:04.700202    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh" podUID="f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" containerName="maas-api" containerID="cri-o://a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d" gracePeriod=30
Sep 10 15:53:04.962570 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:04.962509    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:53:05.032737 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.032706    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-openshift-service-ca\") pod \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") "
Sep 10 15:53:05.032935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.032790    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-metrics-certs\") pod \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") "
Sep 10 15:53:05.032935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.032850    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-maas-api-tls\") pod \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") "
Sep 10 15:53:05.032935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.032876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt5jz\" (UniqueName: \"kubernetes.io/projected/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-kube-api-access-kt5jz\") pod \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\" (UID: \"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4\") "
Sep 10 15:53:05.033165 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.033144    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" (UID: "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:53:05.035095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.035068    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" (UID: "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:53:05.035095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.035085    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" (UID: "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:53:05.035285 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.035113    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-kube-api-access-kt5jz" (OuterVolumeSpecName: "kube-api-access-kt5jz") pod "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" (UID: "f2428ae1-bfd1-41ff-9f6f-df912cfee9b4"). InnerVolumeSpecName "kube-api-access-kt5jz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:53:05.134439 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.134402    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:53:05.134439 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.134430    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:53:05.134439 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.134441    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:53:05.134793 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.134452    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt5jz\" (UniqueName: \"kubernetes.io/projected/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4-kube-api-access-kt5jz\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:53:05.399327 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.399297    2577 generic.go:356] "Generic (PLEG): container finished" podID="f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" containerID="a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d" exitCode=0
Sep 10 15:53:05.399530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.399346    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"
Sep 10 15:53:05.399530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.399383    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh" event={"ID":"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4","Type":"ContainerDied","Data":"a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d"}
Sep 10 15:53:05.399530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.399438    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh" event={"ID":"f2428ae1-bfd1-41ff-9f6f-df912cfee9b4","Type":"ContainerDied","Data":"60092af08eada0115123d437bf2158588f454a7a879b982b374ea01633daafa7"}
Sep 10 15:53:05.399530 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.399466    2577 scope.go:117] "RemoveContainer" containerID="a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d"
Sep 10 15:53:05.416642 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.416624    2577 scope.go:117] "RemoveContainer" containerID="a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d"
Sep 10 15:53:05.416963 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:53:05.416943    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d\": container with ID starting with a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d not found: ID does not exist" containerID="a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d"
Sep 10 15:53:05.417043 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.416970    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d"} err="failed to get container status \"a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d\": rpc error: code = NotFound desc = could not find container \"a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d\": container with ID starting with a5c045ba9579f0d6e815208938536ae55e3251219e1e20587a0e4a7a09916b8d not found: ID does not exist"
Sep 10 15:53:05.425661 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.425641    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"]
Sep 10 15:53:05.429666 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:05.429645    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-c1384d3d-5f67bc79b-nc9mh"]
Sep 10 15:53:06.908102 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:53:06.908068    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" path="/var/lib/kubelet/pods/f2428ae1-bfd1-41ff-9f6f-df912cfee9b4/volumes"
Sep 10 15:54:13.134365 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:13.134327    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"]
Sep 10 15:54:28.154836 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.154764    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9" podUID="2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" containerName="main" containerID="cri-o://b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74" gracePeriod=45
Sep 10 15:54:28.405099 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.405037    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:54:28.537628 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.537596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kc5kj\" (UniqueName: \"kubernetes.io/projected/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-kube-api-access-kc5kj\") pod \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") "
Sep 10 15:54:28.537818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.537648    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-dshm\") pod \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") "
Sep 10 15:54:28.537818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.537665    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-home\") pod \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") "
Sep 10 15:54:28.537818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.537693    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-model-cache\") pod \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") "
Sep 10 15:54:28.537818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.537811    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tmp-dir\") pod \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") "
Sep 10 15:54:28.538028 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.537838    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tls-certs\") pod \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\" (UID: \"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3\") "
Sep 10 15:54:28.538028 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.537922    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-home" (OuterVolumeSpecName: "home") pod "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" (UID: "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:54:28.538138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.538009    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-model-cache" (OuterVolumeSpecName: "model-cache") pod "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" (UID: "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:54:28.538138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.538075    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" (UID: "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:54:28.538138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.538102    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-home\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:54:28.538138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.538123    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-model-cache\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:54:28.540034 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.540015    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-dshm" (OuterVolumeSpecName: "dshm") pod "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" (UID: "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:54:28.540110 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.540077    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" (UID: "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:54:28.540163 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.540143    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-kube-api-access-kc5kj" (OuterVolumeSpecName: "kube-api-access-kc5kj") pod "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" (UID: "2f6c680e-eced-4a1c-9a33-a9e71c9da2e3"). InnerVolumeSpecName "kube-api-access-kc5kj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:54:28.639307 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.639281    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tmp-dir\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:54:28.639307 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.639303    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-tls-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:54:28.639464 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.639314    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kc5kj\" (UniqueName: \"kubernetes.io/projected/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-kube-api-access-kc5kj\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:54:28.639464 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.639323    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3-dshm\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:54:28.861552 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.861515    2577 generic.go:356] "Generic (PLEG): container finished" podID="2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" containerID="b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74" exitCode=0
Sep 10 15:54:28.861768 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.861566    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"
Sep 10 15:54:28.861768 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.861605    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9" event={"ID":"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3","Type":"ContainerDied","Data":"b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74"}
Sep 10 15:54:28.861768 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.861661    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9" event={"ID":"2f6c680e-eced-4a1c-9a33-a9e71c9da2e3","Type":"ContainerDied","Data":"c457ae42b1ce1bb61a56386a8a5206804e9ac1373590ee170a52350cf145ac98"}
Sep 10 15:54:28.861768 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.861685    2577 scope.go:117] "RemoveContainer" containerID="b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74"
Sep 10 15:54:28.871897 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.871732    2577 scope.go:117] "RemoveContainer" containerID="b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74"
Sep 10 15:54:28.872107 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:54:28.872085    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74\": container with ID starting with b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74 not found: ID does not exist" containerID="b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74"
Sep 10 15:54:28.872225 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.872112    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74"} err="failed to get container status \"b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74\": rpc error: code = NotFound desc = could not find container \"b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74\": container with ID starting with b7311ea506503306e3adbc107567c94970814490a57d2e3fc620da9b78351b74 not found: ID does not exist"
Sep 10 15:54:28.882983 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.882957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"]
Sep 10 15:54:28.885247 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.885223    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/auth-test-model-6f8c71-kserve-577fbb74ff-xcrz9"]
Sep 10 15:54:28.910141 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:28.910110    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" path="/var/lib/kubelet/pods/2f6c680e-eced-4a1c-9a33-a9e71c9da2e3/volumes"
Sep 10 15:54:49.200311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.200274    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"]
Sep 10 15:54:49.200950 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.200896    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" containerName="maas-api"
Sep 10 15:54:49.200950 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.200912    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" containerName="maas-api"
Sep 10 15:54:49.200950 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.200940    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" containerName="main"
Sep 10 15:54:49.200950 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.200948    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" containerName="main"
Sep 10 15:54:49.201190 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.201044    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2428ae1-bfd1-41ff-9f6f-df912cfee9b4" containerName="maas-api"
Sep 10 15:54:49.201190 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.201070    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f6c680e-eced-4a1c-9a33-a9e71c9da2e3" containerName="main"
Sep 10 15:54:49.205807 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.205782    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.208311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.208290    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-a653ab-serving-cert\""
Sep 10 15:54:49.210122 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.210096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"]
Sep 10 15:54:49.218202 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.218182    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733e3402-220d-43a1-9ffb-4f954e8c96b4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.218314 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.218215    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-metrics-certs\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.218370 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.218341    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.218422 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.218398    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfxvm\" (UniqueName: \"kubernetes.io/projected/733e3402-220d-43a1-9ffb-4f954e8c96b4-kube-api-access-hfxvm\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.319772 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.319721    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.319951 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.319800    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfxvm\" (UniqueName: \"kubernetes.io/projected/733e3402-220d-43a1-9ffb-4f954e8c96b4-kube-api-access-hfxvm\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.319951 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.319848    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733e3402-220d-43a1-9ffb-4f954e8c96b4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.319951 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:54:49.319855    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-serving-cert: secret "maas-api-e2e-api-a-a653ab-serving-cert" not found
Sep 10 15:54:49.319951 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.319883    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-metrics-certs\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.319951 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:54:49.319929    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls podName:733e3402-220d-43a1-9ffb-4f954e8c96b4 nodeName:}" failed. No retries permitted until 2026-09-10 15:54:49.819908842 +0000 UTC m=+1333.434035737 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls") pod "maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2" (UID: "733e3402-220d-43a1-9ffb-4f954e8c96b4") : secret "maas-api-e2e-api-a-a653ab-serving-cert" not found
Sep 10 15:54:49.320486 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.320463    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733e3402-220d-43a1-9ffb-4f954e8c96b4-openshift-service-ca\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.322569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.322543    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-metrics-certs\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.330458 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.330437    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfxvm\" (UniqueName: \"kubernetes.io/projected/733e3402-220d-43a1-9ffb-4f954e8c96b4-kube-api-access-hfxvm\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.823358 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.823328    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.825981 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.825956    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls\") pod \"maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:49.844588 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.844565    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"]
Sep 10 15:54:49.848402 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.848383    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:49.851024 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.851002    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-a653ab\""
Sep 10 15:54:49.851117 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.851006    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-a653ab-dockercfg-8t9hg\""
Sep 10 15:54:49.860244 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.860222    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"]
Sep 10 15:54:49.924033 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.924009    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsc9w\" (UniqueName: \"kubernetes.io/projected/20fab77a-3f5d-456a-af3c-00227461c2ae-kube-api-access-fsc9w\") pod \"payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:49.924159 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:49.924095    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20fab77a-3f5d-456a-af3c-00227461c2ae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:50.024463 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.024425    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsc9w\" (UniqueName: \"kubernetes.io/projected/20fab77a-3f5d-456a-af3c-00227461c2ae-kube-api-access-fsc9w\") pod \"payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:50.024617 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.024526    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20fab77a-3f5d-456a-af3c-00227461c2ae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:50.025200 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.025181    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20fab77a-3f5d-456a-af3c-00227461c2ae-plugins-config-volume\") pod \"payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:50.032270 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.032240    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsc9w\" (UniqueName: \"kubernetes.io/projected/20fab77a-3f5d-456a-af3c-00227461c2ae-kube-api-access-fsc9w\") pod \"payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") " pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:50.119150 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.119130    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:50.160320 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.160286    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:50.391225 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.391187    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"]
Sep 10 15:54:50.392872 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:54:50.392844    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod733e3402_220d_43a1_9ffb_4f954e8c96b4.slice/crio-1750faca026a4059773bedf458846f16741e19505b8eae7e7982e7fcc8f05910 WatchSource:0}: Error finding container 1750faca026a4059773bedf458846f16741e19505b8eae7e7982e7fcc8f05910: Status 404 returned error can't find the container with id 1750faca026a4059773bedf458846f16741e19505b8eae7e7982e7fcc8f05910
Sep 10 15:54:50.445120 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.445099    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"]
Sep 10 15:54:50.446343 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:54:50.446317    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fab77a_3f5d_456a_af3c_00227461c2ae.slice/crio-c9e34ac8aa79b4b6bc86a27cde5464dfee109f161445f4e9081e9a4207573306 WatchSource:0}: Error finding container c9e34ac8aa79b4b6bc86a27cde5464dfee109f161445f4e9081e9a4207573306: Status 404 returned error can't find the container with id c9e34ac8aa79b4b6bc86a27cde5464dfee109f161445f4e9081e9a4207573306
Sep 10 15:54:50.987822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.987788    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7" event={"ID":"20fab77a-3f5d-456a-af3c-00227461c2ae","Type":"ContainerStarted","Data":"8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2"}
Sep 10 15:54:50.988071 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.988054    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:54:50.988183 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.988168    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7" event={"ID":"20fab77a-3f5d-456a-af3c-00227461c2ae","Type":"ContainerStarted","Data":"c9e34ac8aa79b4b6bc86a27cde5464dfee109f161445f4e9081e9a4207573306"}
Sep 10 15:54:50.990331 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.990302    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2" event={"ID":"733e3402-220d-43a1-9ffb-4f954e8c96b4","Type":"ContainerStarted","Data":"46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8"}
Sep 10 15:54:50.990455 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.990358    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:54:50.990455 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:50.990381    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2" event={"ID":"733e3402-220d-43a1-9ffb-4f954e8c96b4","Type":"ContainerStarted","Data":"1750faca026a4059773bedf458846f16741e19505b8eae7e7982e7fcc8f05910"}
Sep 10 15:54:51.006224 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:51.006175    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7" podStartSLOduration=2.006162297 podStartE2EDuration="2.006162297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:51.003018224 +0000 UTC m=+1334.617145128" watchObservedRunningTime="2026-09-10 15:54:51.006162297 +0000 UTC m=+1334.620289192"
Sep 10 15:54:51.020649 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:51.020610    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2" podStartSLOduration=1.608744628 podStartE2EDuration="2.020597951s" totalImagesPullingTime="411.853323ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:51.018307941 +0000 UTC m=+1334.632434845" watchObservedRunningTime="2026-09-10 15:54:51.020597951 +0000 UTC m=+1334.634724855"
Sep 10 15:54:57.004199 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:54:57.004172    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:55:03.006206 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:03.006121    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:55:16.714674 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:16.714636    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-8177-9d468d"]
Sep 10 15:55:16.718317 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:16.718294    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:16.720974 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:16.720950    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-tpk2l\""
Sep 10 15:55:16.723682 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:16.723658    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8177-9d468d"]
Sep 10 15:55:16.846574 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:16.846541    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bvlf\" (UniqueName: \"kubernetes.io/projected/44900934-69f7-4527-a886-7829ce72b0e9-kube-api-access-9bvlf\") pod \"mt-curl-8177-9d468d\" (UID: \"44900934-69f7-4527-a886-7829ce72b0e9\") " pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:16.947595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:16.947560    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bvlf\" (UniqueName: \"kubernetes.io/projected/44900934-69f7-4527-a886-7829ce72b0e9-kube-api-access-9bvlf\") pod \"mt-curl-8177-9d468d\" (UID: \"44900934-69f7-4527-a886-7829ce72b0e9\") " pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:16.955581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:16.955557    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bvlf\" (UniqueName: \"kubernetes.io/projected/44900934-69f7-4527-a886-7829ce72b0e9-kube-api-access-9bvlf\") pod \"mt-curl-8177-9d468d\" (UID: \"44900934-69f7-4527-a886-7829ce72b0e9\") " pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:17.029300 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:17.029229    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:17.312710 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:17.312686    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8177-9d468d"]
Sep 10 15:55:17.313589 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:55:17.313562    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44900934_69f7_4527_a886_7829ce72b0e9.slice/crio-5159c9ee804389f3ae0b39b8b2ad6ebb0ecbca6208c6c826b48d8fbad9120e1a WatchSource:0}: Error finding container 5159c9ee804389f3ae0b39b8b2ad6ebb0ecbca6208c6c826b48d8fbad9120e1a: Status 404 returned error can't find the container with id 5159c9ee804389f3ae0b39b8b2ad6ebb0ecbca6208c6c826b48d8fbad9120e1a
Sep 10 15:55:18.151412 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:18.151382    2577 generic.go:356] "Generic (PLEG): container finished" podID="44900934-69f7-4527-a886-7829ce72b0e9" containerID="31db1814fa2c4412c820a435a6a8876776f3f0de96b7337fd61765aa3d62523b" exitCode=0
Sep 10 15:55:18.151813 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:18.151472    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8177-9d468d" event={"ID":"44900934-69f7-4527-a886-7829ce72b0e9","Type":"ContainerDied","Data":"31db1814fa2c4412c820a435a6a8876776f3f0de96b7337fd61765aa3d62523b"}
Sep 10 15:55:18.151813 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:18.151527    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8177-9d468d" event={"ID":"44900934-69f7-4527-a886-7829ce72b0e9","Type":"ContainerStarted","Data":"5159c9ee804389f3ae0b39b8b2ad6ebb0ecbca6208c6c826b48d8fbad9120e1a"}
Sep 10 15:55:19.291158 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.291133    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:19.325863 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.325834    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-8177-9d468d_44900934-69f7-4527-a886-7829ce72b0e9/mt-curl-8177-9d468d/0.log"
Sep 10 15:55:19.347030 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.347008    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-8177-9d468d"]
Sep 10 15:55:19.352841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.352817    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-8177-9d468d"]
Sep 10 15:55:19.369587 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.369564    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bvlf\" (UniqueName: \"kubernetes.io/projected/44900934-69f7-4527-a886-7829ce72b0e9-kube-api-access-9bvlf\") pod \"44900934-69f7-4527-a886-7829ce72b0e9\" (UID: \"44900934-69f7-4527-a886-7829ce72b0e9\") "
Sep 10 15:55:19.371667 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.371637    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44900934-69f7-4527-a886-7829ce72b0e9-kube-api-access-9bvlf" (OuterVolumeSpecName: "kube-api-access-9bvlf") pod "44900934-69f7-4527-a886-7829ce72b0e9" (UID: "44900934-69f7-4527-a886-7829ce72b0e9"). InnerVolumeSpecName "kube-api-access-9bvlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:19.470834 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.470745    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bvlf\" (UniqueName: \"kubernetes.io/projected/44900934-69f7-4527-a886-7829ce72b0e9-kube-api-access-9bvlf\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:55:19.492883 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.492856    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-8177-8a9499"]
Sep 10 15:55:19.493271 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.493255    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44900934-69f7-4527-a886-7829ce72b0e9" containerName="mt-curl-8177-9d468d"
Sep 10 15:55:19.493271 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.493270    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="44900934-69f7-4527-a886-7829ce72b0e9" containerName="mt-curl-8177-9d468d"
Sep 10 15:55:19.493422 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.493327    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="44900934-69f7-4527-a886-7829ce72b0e9" containerName="mt-curl-8177-9d468d"
Sep 10 15:55:19.496313 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.496295    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-8a9499"
Sep 10 15:55:19.503033 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.503014    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8177-8a9499"]
Sep 10 15:55:19.571580 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.571556    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4v26\" (UniqueName: \"kubernetes.io/projected/f5e60c30-307d-4341-9d64-003471f337a0-kube-api-access-j4v26\") pod \"mt-curl-8177-8a9499\" (UID: \"f5e60c30-307d-4341-9d64-003471f337a0\") " pod="opendatahub/mt-curl-8177-8a9499"
Sep 10 15:55:19.672482 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.672456    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4v26\" (UniqueName: \"kubernetes.io/projected/f5e60c30-307d-4341-9d64-003471f337a0-kube-api-access-j4v26\") pod \"mt-curl-8177-8a9499\" (UID: \"f5e60c30-307d-4341-9d64-003471f337a0\") " pod="opendatahub/mt-curl-8177-8a9499"
Sep 10 15:55:19.679698 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.679669    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4v26\" (UniqueName: \"kubernetes.io/projected/f5e60c30-307d-4341-9d64-003471f337a0-kube-api-access-j4v26\") pod \"mt-curl-8177-8a9499\" (UID: \"f5e60c30-307d-4341-9d64-003471f337a0\") " pod="opendatahub/mt-curl-8177-8a9499"
Sep 10 15:55:19.807714 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:19.807653    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-8a9499"
Sep 10 15:55:20.080867 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:20.080838    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-8177-8a9499"]
Sep 10 15:55:20.082681 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:55:20.082654    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf5e60c30_307d_4341_9d64_003471f337a0.slice/crio-4939b1265d2a8973d299755c456a3c9b3de13ac82e3fbc795e8e1bf953cb51d4 WatchSource:0}: Error finding container 4939b1265d2a8973d299755c456a3c9b3de13ac82e3fbc795e8e1bf953cb51d4: Status 404 returned error can't find the container with id 4939b1265d2a8973d299755c456a3c9b3de13ac82e3fbc795e8e1bf953cb51d4
Sep 10 15:55:20.169884 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:20.169856    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:20.170027 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:20.169887    2577 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5159c9ee804389f3ae0b39b8b2ad6ebb0ecbca6208c6c826b48d8fbad9120e1a"
Sep 10 15:55:20.171163 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:20.171137    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8177-8a9499" event={"ID":"f5e60c30-307d-4341-9d64-003471f337a0","Type":"ContainerStarted","Data":"4939b1265d2a8973d299755c456a3c9b3de13ac82e3fbc795e8e1bf953cb51d4"}
Sep 10 15:55:20.180034 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:55:20.180010    2577 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-8177-9d468d\" is forbidden: User \"system:node:ip-10-0-139-11.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-139-11.ec2.internal' and this object" podUID="44900934-69f7-4527-a886-7829ce72b0e9" pod="opendatahub/mt-curl-8177-9d468d"
Sep 10 15:55:20.908003 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:20.907971    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44900934-69f7-4527-a886-7829ce72b0e9" path="/var/lib/kubelet/pods/44900934-69f7-4527-a886-7829ce72b0e9/volumes"
Sep 10 15:55:21.178427 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:21.178347    2577 generic.go:356] "Generic (PLEG): container finished" podID="f5e60c30-307d-4341-9d64-003471f337a0" containerID="a413a0c2f61a499a2da8ae31e96cbc6292df2463c4bdcfcbde29bb0d60597a09" exitCode=0
Sep 10 15:55:21.178574 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:21.178433    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-8177-8a9499" event={"ID":"f5e60c30-307d-4341-9d64-003471f337a0","Type":"ContainerDied","Data":"a413a0c2f61a499a2da8ae31e96cbc6292df2463c4bdcfcbde29bb0d60597a09"}
Sep 10 15:55:22.325910 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.325885    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-8a9499"
Sep 10 15:55:22.344301 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.344282    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-8177-8a9499_f5e60c30-307d-4341-9d64-003471f337a0/mt-curl-8177-8a9499/0.log"
Sep 10 15:55:22.367192 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.367167    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-8177-8a9499"]
Sep 10 15:55:22.370578 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.370554    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-8177-8a9499"]
Sep 10 15:55:22.394429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.394405    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j4v26\" (UniqueName: \"kubernetes.io/projected/f5e60c30-307d-4341-9d64-003471f337a0-kube-api-access-j4v26\") pod \"f5e60c30-307d-4341-9d64-003471f337a0\" (UID: \"f5e60c30-307d-4341-9d64-003471f337a0\") "
Sep 10 15:55:22.396534 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.396512    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5e60c30-307d-4341-9d64-003471f337a0-kube-api-access-j4v26" (OuterVolumeSpecName: "kube-api-access-j4v26") pod "f5e60c30-307d-4341-9d64-003471f337a0" (UID: "f5e60c30-307d-4341-9d64-003471f337a0"). InnerVolumeSpecName "kube-api-access-j4v26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:22.496160 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.496091    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4v26\" (UniqueName: \"kubernetes.io/projected/f5e60c30-307d-4341-9d64-003471f337a0-kube-api-access-j4v26\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:55:22.909924 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:22.909887    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f5e60c30-307d-4341-9d64-003471f337a0" path="/var/lib/kubelet/pods/f5e60c30-307d-4341-9d64-003471f337a0/volumes"
Sep 10 15:55:23.195393 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:23.195312    2577 scope.go:117] "RemoveContainer" containerID="a413a0c2f61a499a2da8ae31e96cbc6292df2463c4bdcfcbde29bb0d60597a09"
Sep 10 15:55:23.195738 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:23.195711    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-8177-8a9499"
Sep 10 15:55:32.840196 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.840160    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"]
Sep 10 15:55:32.840888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.840786    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f5e60c30-307d-4341-9d64-003471f337a0" containerName="mt-curl-8177-8a9499"
Sep 10 15:55:32.840888 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.840813    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="f5e60c30-307d-4341-9d64-003471f337a0" containerName="mt-curl-8177-8a9499"
Sep 10 15:55:32.841074 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.840934    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="f5e60c30-307d-4341-9d64-003471f337a0" containerName="mt-curl-8177-8a9499"
Sep 10 15:55:32.845281 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.845261    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:32.848781 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.848689    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"kube-root-ca.crt\""
Sep 10 15:55:32.848781 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.848690    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"default-dockercfg-g7czv\""
Sep 10 15:55:32.848966 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.848690    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"rate-test-model-6f8c71-kserve-self-signed-certs\""
Sep 10 15:55:32.849032 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.848982    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-6f8c71\"/\"openshift-service-ca.crt\""
Sep 10 15:55:32.856110 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.856085    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"]
Sep 10 15:55:32.984833 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.984796    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-home\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:32.984991 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.984849    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwzsf\" (UniqueName: \"kubernetes.io/projected/9a1891c1-4602-4f3f-b195-5d6056f3a771-kube-api-access-jwzsf\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:32.984991 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.984894    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1891c1-4602-4f3f-b195-5d6056f3a771-tls-certs\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:32.984991 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.984924    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-dshm\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:32.984991 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.984981    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-tmp-dir\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:32.985168 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:32.985041    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-model-cache\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086185 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086155    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-home\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086349 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086188    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwzsf\" (UniqueName: \"kubernetes.io/projected/9a1891c1-4602-4f3f-b195-5d6056f3a771-kube-api-access-jwzsf\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086349 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086212    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1891c1-4602-4f3f-b195-5d6056f3a771-tls-certs\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086349 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086237    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-dshm\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086349 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086267    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-tmp-dir\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086349 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086296    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-model-cache\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086599 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086550    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-home\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086745 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086725    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-model-cache\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.086853 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.086724    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-tmp-dir\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.088923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.088890    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-dshm\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.089294 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.089263    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1891c1-4602-4f3f-b195-5d6056f3a771-tls-certs\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.094719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.094661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwzsf\" (UniqueName: \"kubernetes.io/projected/9a1891c1-4602-4f3f-b195-5d6056f3a771-kube-api-access-jwzsf\") pod \"rate-test-model-6f8c71-kserve-77589548bc-z6fpp\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") " pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.157379 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.157343    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:33.433399 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:33.433375    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"]
Sep 10 15:55:33.434626 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:55:33.434601    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a1891c1_4602_4f3f_b195_5d6056f3a771.slice/crio-99f8402ea2fdea4e3338792f8d4f110e43d6d365f7af04d564ed149c173be61e WatchSource:0}: Error finding container 99f8402ea2fdea4e3338792f8d4f110e43d6d365f7af04d564ed149c173be61e: Status 404 returned error can't find the container with id 99f8402ea2fdea4e3338792f8d4f110e43d6d365f7af04d564ed149c173be61e
Sep 10 15:55:34.260569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:34.260535    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp" event={"ID":"9a1891c1-4602-4f3f-b195-5d6056f3a771","Type":"ContainerStarted","Data":"8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf"}
Sep 10 15:55:34.260569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:34.260576    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp" event={"ID":"9a1891c1-4602-4f3f-b195-5d6056f3a771","Type":"ContainerStarted","Data":"99f8402ea2fdea4e3338792f8d4f110e43d6d365f7af04d564ed149c173be61e"}
Sep 10 15:55:34.275702 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:34.275653    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp" podStartSLOduration=2.275634674 podStartE2EDuration="2.275634674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:34.274961433 +0000 UTC m=+1377.889088354" watchObservedRunningTime="2026-09-10 15:55:34.275634674 +0000 UTC m=+1377.889761579"
Sep 10 15:55:43.158141 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:43.158109    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:43.158525 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:43.158175    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:43.170829 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:43.170806    2577 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:55:43.322705 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:55:43.322678    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:56:00.742565 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:00.742534    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"]
Sep 10 15:56:00.743152 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:00.742797    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2" podUID="733e3402-220d-43a1-9ffb-4f954e8c96b4" containerName="maas-api" containerID="cri-o://46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8" gracePeriod=30
Sep 10 15:56:00.804740 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:00.804705    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"]
Sep 10 15:56:00.804996 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:00.804967    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7" podUID="20fab77a-3f5d-456a-af3c-00227461c2ae" containerName="payload-processing" containerID="cri-o://8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2" gracePeriod=30
Sep 10 15:56:01.053294 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.053271    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:56:01.085102 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.085080    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:56:01.128270 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128237    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733e3402-220d-43a1-9ffb-4f954e8c96b4-openshift-service-ca\") pod \"733e3402-220d-43a1-9ffb-4f954e8c96b4\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") "
Sep 10 15:56:01.128441 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128292    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hfxvm\" (UniqueName: \"kubernetes.io/projected/733e3402-220d-43a1-9ffb-4f954e8c96b4-kube-api-access-hfxvm\") pod \"733e3402-220d-43a1-9ffb-4f954e8c96b4\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") "
Sep 10 15:56:01.128441 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20fab77a-3f5d-456a-af3c-00227461c2ae-plugins-config-volume\") pod \"20fab77a-3f5d-456a-af3c-00227461c2ae\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") "
Sep 10 15:56:01.128441 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128414    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-metrics-certs\") pod \"733e3402-220d-43a1-9ffb-4f954e8c96b4\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") "
Sep 10 15:56:01.128613 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128451    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls\") pod \"733e3402-220d-43a1-9ffb-4f954e8c96b4\" (UID: \"733e3402-220d-43a1-9ffb-4f954e8c96b4\") "
Sep 10 15:56:01.128613 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128485    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fsc9w\" (UniqueName: \"kubernetes.io/projected/20fab77a-3f5d-456a-af3c-00227461c2ae-kube-api-access-fsc9w\") pod \"20fab77a-3f5d-456a-af3c-00227461c2ae\" (UID: \"20fab77a-3f5d-456a-af3c-00227461c2ae\") "
Sep 10 15:56:01.128768 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128719    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/733e3402-220d-43a1-9ffb-4f954e8c96b4-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "733e3402-220d-43a1-9ffb-4f954e8c96b4" (UID: "733e3402-220d-43a1-9ffb-4f954e8c96b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:01.128841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fab77a-3f5d-456a-af3c-00227461c2ae-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20fab77a-3f5d-456a-af3c-00227461c2ae" (UID: "20fab77a-3f5d-456a-af3c-00227461c2ae"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:01.128841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.128832    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/733e3402-220d-43a1-9ffb-4f954e8c96b4-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:01.130725 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.130692    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "733e3402-220d-43a1-9ffb-4f954e8c96b4" (UID: "733e3402-220d-43a1-9ffb-4f954e8c96b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:01.130866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.130738    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fab77a-3f5d-456a-af3c-00227461c2ae-kube-api-access-fsc9w" (OuterVolumeSpecName: "kube-api-access-fsc9w") pod "20fab77a-3f5d-456a-af3c-00227461c2ae" (UID: "20fab77a-3f5d-456a-af3c-00227461c2ae"). InnerVolumeSpecName "kube-api-access-fsc9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:01.131177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.131163    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/733e3402-220d-43a1-9ffb-4f954e8c96b4-kube-api-access-hfxvm" (OuterVolumeSpecName: "kube-api-access-hfxvm") pod "733e3402-220d-43a1-9ffb-4f954e8c96b4" (UID: "733e3402-220d-43a1-9ffb-4f954e8c96b4"). InnerVolumeSpecName "kube-api-access-hfxvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:01.131222 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.131181    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "733e3402-220d-43a1-9ffb-4f954e8c96b4" (UID: "733e3402-220d-43a1-9ffb-4f954e8c96b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:01.229449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.229415    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfxvm\" (UniqueName: \"kubernetes.io/projected/733e3402-220d-43a1-9ffb-4f954e8c96b4-kube-api-access-hfxvm\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:01.229449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.229443    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20fab77a-3f5d-456a-af3c-00227461c2ae-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:01.229449 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.229453    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:01.229684 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.229461    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/733e3402-220d-43a1-9ffb-4f954e8c96b4-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:01.229684 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.229469    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fsc9w\" (UniqueName: \"kubernetes.io/projected/20fab77a-3f5d-456a-af3c-00227461c2ae-kube-api-access-fsc9w\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:01.411669 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.411636    2577 generic.go:356] "Generic (PLEG): container finished" podID="20fab77a-3f5d-456a-af3c-00227461c2ae" containerID="8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2" exitCode=0
Sep 10 15:56:01.411873 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.411680    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"
Sep 10 15:56:01.411873 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.411717    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7" event={"ID":"20fab77a-3f5d-456a-af3c-00227461c2ae","Type":"ContainerDied","Data":"8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2"}
Sep 10 15:56:01.411873 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.411796    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7" event={"ID":"20fab77a-3f5d-456a-af3c-00227461c2ae","Type":"ContainerDied","Data":"c9e34ac8aa79b4b6bc86a27cde5464dfee109f161445f4e9081e9a4207573306"}
Sep 10 15:56:01.411873 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.411824    2577 scope.go:117] "RemoveContainer" containerID="8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2"
Sep 10 15:56:01.413209 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.413178    2577 generic.go:356] "Generic (PLEG): container finished" podID="733e3402-220d-43a1-9ffb-4f954e8c96b4" containerID="46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8" exitCode=0
Sep 10 15:56:01.413338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.413224    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"
Sep 10 15:56:01.413338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.413265    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2" event={"ID":"733e3402-220d-43a1-9ffb-4f954e8c96b4","Type":"ContainerDied","Data":"46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8"}
Sep 10 15:56:01.413338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.413306    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2" event={"ID":"733e3402-220d-43a1-9ffb-4f954e8c96b4","Type":"ContainerDied","Data":"1750faca026a4059773bedf458846f16741e19505b8eae7e7982e7fcc8f05910"}
Sep 10 15:56:01.421644 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.421567    2577 scope.go:117] "RemoveContainer" containerID="8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2"
Sep 10 15:56:01.421976 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:56:01.421912    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2\": container with ID starting with 8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2 not found: ID does not exist" containerID="8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2"
Sep 10 15:56:01.421976 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.421948    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2"} err="failed to get container status \"8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2\": rpc error: code = NotFound desc = could not find container \"8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2\": container with ID starting with 8f1aa7a3a15a54b5bcd9cf97b979bd42d30c4564472240a56dee49391fa0deb2 not found: ID does not exist"
Sep 10 15:56:01.421976 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.421972    2577 scope.go:117] "RemoveContainer" containerID="46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8"
Sep 10 15:56:01.431095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.431077    2577 scope.go:117] "RemoveContainer" containerID="46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8"
Sep 10 15:56:01.431385 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:56:01.431362    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8\": container with ID starting with 46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8 not found: ID does not exist" containerID="46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8"
Sep 10 15:56:01.431447 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.431391    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8"} err="failed to get container status \"46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8\": rpc error: code = NotFound desc = could not find container \"46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8\": container with ID starting with 46a930abb698587e2f014d350549b7a9962c79e74da0943d12b40026662ff5f8 not found: ID does not exist"
Sep 10 15:56:01.437040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.437019    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"]
Sep 10 15:56:01.439629 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.439609    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-a653ab-7bb6dd6ff9-gf2w7"]
Sep 10 15:56:01.448053 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.448027    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"]
Sep 10 15:56:01.451345 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:01.451326    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-a653ab-66b7c8496c-82wk2"]
Sep 10 15:56:02.908093 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:02.908064    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20fab77a-3f5d-456a-af3c-00227461c2ae" path="/var/lib/kubelet/pods/20fab77a-3f5d-456a-af3c-00227461c2ae/volumes"
Sep 10 15:56:02.908474 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:02.908440    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="733e3402-220d-43a1-9ffb-4f954e8c96b4" path="/var/lib/kubelet/pods/733e3402-220d-43a1-9ffb-4f954e8c96b4/volumes"
Sep 10 15:56:41.006565 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.006534    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"]
Sep 10 15:56:41.007047 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.006805    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb" podUID="8da59559-f754-4b63-bd70-b0c0fe83c6a5" containerName="maas-api" containerID="cri-o://7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be" gracePeriod=30
Sep 10 15:56:41.065817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.065786    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"]
Sep 10 15:56:41.066100 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.066071    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt" podUID="acbfcb1b-4c71-4575-a4df-976e4de543c8" containerName="payload-pre-processing" containerID="cri-o://2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0" gracePeriod=30
Sep 10 15:56:41.332842 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.332820    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:56:41.346342 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.346319    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:56:41.381039 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381019    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbfcb1b-4c71-4575-a4df-976e4de543c8-plugins-config-volume\") pod \"acbfcb1b-4c71-4575-a4df-976e4de543c8\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") "
Sep 10 15:56:41.381188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381072    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nx6fs\" (UniqueName: \"kubernetes.io/projected/acbfcb1b-4c71-4575-a4df-976e4de543c8-kube-api-access-nx6fs\") pod \"acbfcb1b-4c71-4575-a4df-976e4de543c8\" (UID: \"acbfcb1b-4c71-4575-a4df-976e4de543c8\") "
Sep 10 15:56:41.381188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381111    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-metrics-certs\") pod \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") "
Sep 10 15:56:41.381188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381156    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dws2g\" (UniqueName: \"kubernetes.io/projected/8da59559-f754-4b63-bd70-b0c0fe83c6a5-kube-api-access-dws2g\") pod \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") "
Sep 10 15:56:41.381357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381327    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da59559-f754-4b63-bd70-b0c0fe83c6a5-openshift-service-ca\") pod \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") "
Sep 10 15:56:41.381418 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381390    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls\") pod \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\" (UID: \"8da59559-f754-4b63-bd70-b0c0fe83c6a5\") "
Sep 10 15:56:41.381483 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381435    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acbfcb1b-4c71-4575-a4df-976e4de543c8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acbfcb1b-4c71-4575-a4df-976e4de543c8" (UID: "acbfcb1b-4c71-4575-a4df-976e4de543c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:41.381711 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381681    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acbfcb1b-4c71-4575-a4df-976e4de543c8-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.381838 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.381704    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8da59559-f754-4b63-bd70-b0c0fe83c6a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8da59559-f754-4b63-bd70-b0c0fe83c6a5" (UID: "8da59559-f754-4b63-bd70-b0c0fe83c6a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:41.383576 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.383547    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acbfcb1b-4c71-4575-a4df-976e4de543c8-kube-api-access-nx6fs" (OuterVolumeSpecName: "kube-api-access-nx6fs") pod "acbfcb1b-4c71-4575-a4df-976e4de543c8" (UID: "acbfcb1b-4c71-4575-a4df-976e4de543c8"). InnerVolumeSpecName "kube-api-access-nx6fs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:41.383722 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.383699    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8da59559-f754-4b63-bd70-b0c0fe83c6a5" (UID: "8da59559-f754-4b63-bd70-b0c0fe83c6a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:41.383949 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.383927    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8da59559-f754-4b63-bd70-b0c0fe83c6a5" (UID: "8da59559-f754-4b63-bd70-b0c0fe83c6a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:41.384040 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.383962    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8da59559-f754-4b63-bd70-b0c0fe83c6a5-kube-api-access-dws2g" (OuterVolumeSpecName: "kube-api-access-dws2g") pod "8da59559-f754-4b63-bd70-b0c0fe83c6a5" (UID: "8da59559-f754-4b63-bd70-b0c0fe83c6a5"). InnerVolumeSpecName "kube-api-access-dws2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:41.482562 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.482521    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.482562 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.482558    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dws2g\" (UniqueName: \"kubernetes.io/projected/8da59559-f754-4b63-bd70-b0c0fe83c6a5-kube-api-access-dws2g\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.482746 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.482573    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8da59559-f754-4b63-bd70-b0c0fe83c6a5-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.482746 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.482587    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8da59559-f754-4b63-bd70-b0c0fe83c6a5-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.482746 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.482601    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nx6fs\" (UniqueName: \"kubernetes.io/projected/acbfcb1b-4c71-4575-a4df-976e4de543c8-kube-api-access-nx6fs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:41.653899 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.653867    2577 generic.go:356] "Generic (PLEG): container finished" podID="acbfcb1b-4c71-4575-a4df-976e4de543c8" containerID="2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0" exitCode=0
Sep 10 15:56:41.654088 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.653911    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"
Sep 10 15:56:41.654088 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.653966    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt" event={"ID":"acbfcb1b-4c71-4575-a4df-976e4de543c8","Type":"ContainerDied","Data":"2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0"}
Sep 10 15:56:41.654088 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.654021    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt" event={"ID":"acbfcb1b-4c71-4575-a4df-976e4de543c8","Type":"ContainerDied","Data":"16f761815872038784c01b6d798873ca9687e7df1b844aff7f474e6de64f651c"}
Sep 10 15:56:41.654088 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.654043    2577 scope.go:117] "RemoveContainer" containerID="2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0"
Sep 10 15:56:41.655447 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.655420    2577 generic.go:356] "Generic (PLEG): container finished" podID="8da59559-f754-4b63-bd70-b0c0fe83c6a5" containerID="7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be" exitCode=0
Sep 10 15:56:41.655581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.655465    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"
Sep 10 15:56:41.655581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.655503    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb" event={"ID":"8da59559-f754-4b63-bd70-b0c0fe83c6a5","Type":"ContainerDied","Data":"7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be"}
Sep 10 15:56:41.655581 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.655551    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb" event={"ID":"8da59559-f754-4b63-bd70-b0c0fe83c6a5","Type":"ContainerDied","Data":"ff22b7333bf5e846abd6e0e3dad5c9671589cb41046849daa1523b6819ac3484"}
Sep 10 15:56:41.665936 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.665917    2577 scope.go:117] "RemoveContainer" containerID="2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0"
Sep 10 15:56:41.666216 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:56:41.666196    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0\": container with ID starting with 2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0 not found: ID does not exist" containerID="2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0"
Sep 10 15:56:41.666279 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.666225    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0"} err="failed to get container status \"2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0\": rpc error: code = NotFound desc = could not find container \"2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0\": container with ID starting with 2fe2d7295c63158b7212c086134e1c2be7ea63bb640632d3a8ddd44b5c641aa0 not found: ID does not exist"
Sep 10 15:56:41.666279 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.666241    2577 scope.go:117] "RemoveContainer" containerID="7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be"
Sep 10 15:56:41.674839 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.674821    2577 scope.go:117] "RemoveContainer" containerID="7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be"
Sep 10 15:56:41.675087 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:56:41.675070    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be\": container with ID starting with 7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be not found: ID does not exist" containerID="7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be"
Sep 10 15:56:41.675149 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.675093    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be"} err="failed to get container status \"7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be\": rpc error: code = NotFound desc = could not find container \"7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be\": container with ID starting with 7cde7b5c6f0621a74941e2b7197cf0a223aef2ee17e01367e02ef2ab3f0621be not found: ID does not exist"
Sep 10 15:56:41.679366 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.679343    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"]
Sep 10 15:56:41.684719 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.684698    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-8e3dc2-577b844d98-85ctt"]
Sep 10 15:56:41.695378 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.695357    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"]
Sep 10 15:56:41.701429 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:41.701409    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w2-8e3dc2-796f8dc6d9-h6qbb"]
Sep 10 15:56:42.908125 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:42.908092    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8da59559-f754-4b63-bd70-b0c0fe83c6a5" path="/var/lib/kubelet/pods/8da59559-f754-4b63-bd70-b0c0fe83c6a5/volumes"
Sep 10 15:56:42.908495 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:42.908435    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acbfcb1b-4c71-4575-a4df-976e4de543c8" path="/var/lib/kubelet/pods/acbfcb1b-4c71-4575-a4df-976e4de543c8/volumes"
Sep 10 15:56:52.526427 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:52.526394    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"]
Sep 10 15:56:57.919794 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.919740    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"]
Sep 10 15:56:57.920309 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920290    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20fab77a-3f5d-456a-af3c-00227461c2ae" containerName="payload-processing"
Sep 10 15:56:57.920387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920313    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="20fab77a-3f5d-456a-af3c-00227461c2ae" containerName="payload-processing"
Sep 10 15:56:57.920387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920333    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8da59559-f754-4b63-bd70-b0c0fe83c6a5" containerName="maas-api"
Sep 10 15:56:57.920387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920342    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="8da59559-f754-4b63-bd70-b0c0fe83c6a5" containerName="maas-api"
Sep 10 15:56:57.920387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920354    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="733e3402-220d-43a1-9ffb-4f954e8c96b4" containerName="maas-api"
Sep 10 15:56:57.920387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920363    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="733e3402-220d-43a1-9ffb-4f954e8c96b4" containerName="maas-api"
Sep 10 15:56:57.920387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920377    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acbfcb1b-4c71-4575-a4df-976e4de543c8" containerName="payload-pre-processing"
Sep 10 15:56:57.920387 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920385    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="acbfcb1b-4c71-4575-a4df-976e4de543c8" containerName="payload-pre-processing"
Sep 10 15:56:57.920731 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920486    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="8da59559-f754-4b63-bd70-b0c0fe83c6a5" containerName="maas-api"
Sep 10 15:56:57.920731 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920497    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="20fab77a-3f5d-456a-af3c-00227461c2ae" containerName="payload-processing"
Sep 10 15:56:57.920731 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920509    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="acbfcb1b-4c71-4575-a4df-976e4de543c8" containerName="payload-pre-processing"
Sep 10 15:56:57.920731 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.920522    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="733e3402-220d-43a1-9ffb-4f954e8c96b4" containerName="maas-api"
Sep 10 15:56:57.925298 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.925274    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:57.927622 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.927601    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-a85a91-serving-cert\""
Sep 10 15:56:57.941301 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:57.940689    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"]
Sep 10 15:56:58.021034 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.020999    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.021240 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.021203    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf5ng\" (UniqueName: \"kubernetes.io/projected/078262ba-701a-46a9-a951-d258d5fd6c37-kube-api-access-lf5ng\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.021310 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.021261    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-metrics-certs\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.021378 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.021357    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078262ba-701a-46a9-a951-d258d5fd6c37-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.122591 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.122545    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf5ng\" (UniqueName: \"kubernetes.io/projected/078262ba-701a-46a9-a951-d258d5fd6c37-kube-api-access-lf5ng\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.122591 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.122593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-metrics-certs\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.122866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.122632    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078262ba-701a-46a9-a951-d258d5fd6c37-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.122866 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.122686    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.122956 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:56:58.122895    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-serving-cert: secret "maas-api-e2e-ipp-a-a85a91-serving-cert" not found
Sep 10 15:56:58.123006 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:56:58.122985    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls podName:078262ba-701a-46a9-a951-d258d5fd6c37 nodeName:}" failed. No retries permitted until 2026-09-10 15:56:58.622961915 +0000 UTC m=+1462.237088799 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls") pod "maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r" (UID: "078262ba-701a-46a9-a951-d258d5fd6c37") : secret "maas-api-e2e-ipp-a-a85a91-serving-cert" not found
Sep 10 15:56:58.123739 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.123710    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078262ba-701a-46a9-a951-d258d5fd6c37-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.125470 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.125439    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-metrics-certs\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.131970 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.131948    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf5ng\" (UniqueName: \"kubernetes.io/projected/078262ba-701a-46a9-a951-d258d5fd6c37-kube-api-access-lf5ng\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.565856 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.565821    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"]
Sep 10 15:56:58.569400 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.569384    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:58.572675 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.572651    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a85a91-dockercfg-k869s\""
Sep 10 15:56:58.572880 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.572855    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a85a91\""
Sep 10 15:56:58.591570 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.591545    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"]
Sep 10 15:56:58.628003 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.627979    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.628145 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.628046    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgfcr\" (UniqueName: \"kubernetes.io/projected/bce838d5-ea35-4c54-97b1-32d106646e27-kube-api-access-rgfcr\") pod \"payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:58.628145 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.628084    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce838d5-ea35-4c54-97b1-32d106646e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:58.630725 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.630703    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls\") pod \"maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.729457 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.729419    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgfcr\" (UniqueName: \"kubernetes.io/projected/bce838d5-ea35-4c54-97b1-32d106646e27-kube-api-access-rgfcr\") pod \"payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:58.729616 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.729464    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce838d5-ea35-4c54-97b1-32d106646e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:58.730203 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.730178    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce838d5-ea35-4c54-97b1-32d106646e27-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:58.737629 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.737604    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgfcr\" (UniqueName: \"kubernetes.io/projected/bce838d5-ea35-4c54-97b1-32d106646e27-kube-api-access-rgfcr\") pod \"payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:58.791069 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.791029    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"]
Sep 10 15:56:58.791563 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.791529    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt" podUID="7e2307c6-a64d-406a-a3b5-e6ce7992e96f" containerName="maas-api" containerID="cri-o://00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470" gracePeriod=30
Sep 10 15:56:58.838533 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.838507    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:56:58.880610 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:58.880572    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:59.065151 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.065124    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:56:59.125986 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.125957    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"]
Sep 10 15:56:59.126439 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:56:59.126405    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod078262ba_701a_46a9_a951_d258d5fd6c37.slice/crio-cd4a1b2109f2e2fa6ae1e1eead973006aa039ab0c5fc29b5f54ed3c8116454ae WatchSource:0}: Error finding container cd4a1b2109f2e2fa6ae1e1eead973006aa039ab0c5fc29b5f54ed3c8116454ae: Status 404 returned error can't find the container with id cd4a1b2109f2e2fa6ae1e1eead973006aa039ab0c5fc29b5f54ed3c8116454ae
Sep 10 15:56:59.127826 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.127809    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:56:59.132634 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.132614    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7p4hp\" (UniqueName: \"kubernetes.io/projected/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-kube-api-access-7p4hp\") pod \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") "
Sep 10 15:56:59.132724 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.132675    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls\") pod \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") "
Sep 10 15:56:59.132724 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.132710    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-openshift-service-ca\") pod \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") "
Sep 10 15:56:59.132877 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.132745    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-metrics-certs\") pod \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\" (UID: \"7e2307c6-a64d-406a-a3b5-e6ce7992e96f\") "
Sep 10 15:56:59.133131 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.133108    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7e2307c6-a64d-406a-a3b5-e6ce7992e96f" (UID: "7e2307c6-a64d-406a-a3b5-e6ce7992e96f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:59.135000 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.134870    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7e2307c6-a64d-406a-a3b5-e6ce7992e96f" (UID: "7e2307c6-a64d-406a-a3b5-e6ce7992e96f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:59.135120 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.135064    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7e2307c6-a64d-406a-a3b5-e6ce7992e96f" (UID: "7e2307c6-a64d-406a-a3b5-e6ce7992e96f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:56:59.135239 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.135194    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-kube-api-access-7p4hp" (OuterVolumeSpecName: "kube-api-access-7p4hp") pod "7e2307c6-a64d-406a-a3b5-e6ce7992e96f" (UID: "7e2307c6-a64d-406a-a3b5-e6ce7992e96f"). InnerVolumeSpecName "kube-api-access-7p4hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:59.176485 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.176465    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"]
Sep 10 15:56:59.178889 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:56:59.178828    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbce838d5_ea35_4c54_97b1_32d106646e27.slice/crio-970f9f640af8826b89761c7f83e210b116ce652e3b4519c0bb873707772c9a76 WatchSource:0}: Error finding container 970f9f640af8826b89761c7f83e210b116ce652e3b4519c0bb873707772c9a76: Status 404 returned error can't find the container with id 970f9f640af8826b89761c7f83e210b116ce652e3b4519c0bb873707772c9a76
Sep 10 15:56:59.233817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.233788    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7p4hp\" (UniqueName: \"kubernetes.io/projected/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-kube-api-access-7p4hp\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:59.233817 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.233812    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:59.233952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.233821    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:59.233952 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.233830    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7e2307c6-a64d-406a-a3b5-e6ce7992e96f-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:56:59.758678 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.758642    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds" event={"ID":"bce838d5-ea35-4c54-97b1-32d106646e27","Type":"ContainerStarted","Data":"822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e"}
Sep 10 15:56:59.758852 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.758709    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds" event={"ID":"bce838d5-ea35-4c54-97b1-32d106646e27","Type":"ContainerStarted","Data":"970f9f640af8826b89761c7f83e210b116ce652e3b4519c0bb873707772c9a76"}
Sep 10 15:56:59.758919 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.758864    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:56:59.760128 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.760104    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r" event={"ID":"078262ba-701a-46a9-a951-d258d5fd6c37","Type":"ContainerStarted","Data":"cd4a1b2109f2e2fa6ae1e1eead973006aa039ab0c5fc29b5f54ed3c8116454ae"}
Sep 10 15:56:59.761818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.761797    2577 generic.go:356] "Generic (PLEG): container finished" podID="7e2307c6-a64d-406a-a3b5-e6ce7992e96f" containerID="00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470" exitCode=0
Sep 10 15:56:59.761913 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.761891    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"
Sep 10 15:56:59.762046 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.761858    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt" event={"ID":"7e2307c6-a64d-406a-a3b5-e6ce7992e96f","Type":"ContainerDied","Data":"00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470"}
Sep 10 15:56:59.762094 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.762073    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt" event={"ID":"7e2307c6-a64d-406a-a3b5-e6ce7992e96f","Type":"ContainerDied","Data":"a6914ad0ffee6c31c26522c79effcf36d677031b3245208eccef9eadb93a4e3d"}
Sep 10 15:56:59.762134 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.762100    2577 scope.go:117] "RemoveContainer" containerID="00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470"
Sep 10 15:56:59.772306 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.772286    2577 scope.go:117] "RemoveContainer" containerID="00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470"
Sep 10 15:56:59.772603 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:56:59.772586    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470\": container with ID starting with 00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470 not found: ID does not exist" containerID="00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470"
Sep 10 15:56:59.772650 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.772611    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470"} err="failed to get container status \"00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470\": rpc error: code = NotFound desc = could not find container \"00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470\": container with ID starting with 00de435b9a185f87f49accfe684b356aa94a5534b2389501da114670aac8a470 not found: ID does not exist"
Sep 10 15:56:59.778702 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.778657    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds" podStartSLOduration=1.7786419439999999 podStartE2EDuration="1.778641944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:59.775577205 +0000 UTC m=+1463.389704109" watchObservedRunningTime="2026-09-10 15:56:59.778641944 +0000 UTC m=+1463.392768873"
Sep 10 15:56:59.791231 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.791201    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"]
Sep 10 15:56:59.797575 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:56:59.797543    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-761990-56c98cb759-xz4qt"]
Sep 10 15:57:00.770122 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:00.770081    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r" event={"ID":"078262ba-701a-46a9-a951-d258d5fd6c37","Type":"ContainerStarted","Data":"47b50f5abf9cba23fdbfdc07a0712069f3259e861f5129e65b0c0dc8a5594364"}
Sep 10 15:57:00.770611 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:00.770244    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:57:00.790885 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:00.790833    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r" podStartSLOduration=3.201204492 podStartE2EDuration="3.790813346s" totalImagesPullingTime="589.608854ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:00.787069431 +0000 UTC m=+1464.401196336" watchObservedRunningTime="2026-09-10 15:57:00.790813346 +0000 UTC m=+1464.404940251"
Sep 10 15:57:00.908059 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:00.908033    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e2307c6-a64d-406a-a3b5-e6ce7992e96f" path="/var/lib/kubelet/pods/7e2307c6-a64d-406a-a3b5-e6ce7992e96f/volumes"
Sep 10 15:57:06.781734 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:06.781706    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:57:07.546526 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.546486    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp" podUID="9a1891c1-4602-4f3f-b195-5d6056f3a771" containerName="main" containerID="cri-o://8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf" gracePeriod=45
Sep 10 15:57:07.799219 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.799166    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:57:07.812818 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.812790    2577 generic.go:356] "Generic (PLEG): container finished" podID="9a1891c1-4602-4f3f-b195-5d6056f3a771" containerID="8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf" exitCode=0
Sep 10 15:57:07.812923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.812832    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"
Sep 10 15:57:07.812923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.812845    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp" event={"ID":"9a1891c1-4602-4f3f-b195-5d6056f3a771","Type":"ContainerDied","Data":"8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf"}
Sep 10 15:57:07.812923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.812875    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp" event={"ID":"9a1891c1-4602-4f3f-b195-5d6056f3a771","Type":"ContainerDied","Data":"99f8402ea2fdea4e3338792f8d4f110e43d6d365f7af04d564ed149c173be61e"}
Sep 10 15:57:07.812923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.812890    2577 scope.go:117] "RemoveContainer" containerID="8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf"
Sep 10 15:57:07.824552 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.824528    2577 scope.go:117] "RemoveContainer" containerID="8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf"
Sep 10 15:57:07.824922 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:57:07.824900    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf\": container with ID starting with 8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf not found: ID does not exist" containerID="8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf"
Sep 10 15:57:07.825014 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.824931    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf"} err="failed to get container status \"8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf\": rpc error: code = NotFound desc = could not find container \"8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf\": container with ID starting with 8ed75a2ed5eeebd56c1137927e0671caaa3c3e3f1f2053fe888fe1561eb557bf not found: ID does not exist"
Sep 10 15:57:07.920353 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920328    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-home\") pod \"9a1891c1-4602-4f3f-b195-5d6056f3a771\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") "
Sep 10 15:57:07.920595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920386    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1891c1-4602-4f3f-b195-5d6056f3a771-tls-certs\") pod \"9a1891c1-4602-4f3f-b195-5d6056f3a771\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") "
Sep 10 15:57:07.920595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920424    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jwzsf\" (UniqueName: \"kubernetes.io/projected/9a1891c1-4602-4f3f-b195-5d6056f3a771-kube-api-access-jwzsf\") pod \"9a1891c1-4602-4f3f-b195-5d6056f3a771\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") "
Sep 10 15:57:07.920595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920455    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-dshm\") pod \"9a1891c1-4602-4f3f-b195-5d6056f3a771\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") "
Sep 10 15:57:07.920595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920476    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-model-cache\") pod \"9a1891c1-4602-4f3f-b195-5d6056f3a771\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") "
Sep 10 15:57:07.920595 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920511    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-tmp-dir\") pod \"9a1891c1-4602-4f3f-b195-5d6056f3a771\" (UID: \"9a1891c1-4602-4f3f-b195-5d6056f3a771\") "
Sep 10 15:57:07.920898 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920636    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-home" (OuterVolumeSpecName: "home") pod "9a1891c1-4602-4f3f-b195-5d6056f3a771" (UID: "9a1891c1-4602-4f3f-b195-5d6056f3a771"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:07.920898 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920741    2577 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-home\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:57:07.920898 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920796    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-model-cache" (OuterVolumeSpecName: "model-cache") pod "9a1891c1-4602-4f3f-b195-5d6056f3a771" (UID: "9a1891c1-4602-4f3f-b195-5d6056f3a771"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:07.920898 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.920824    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9a1891c1-4602-4f3f-b195-5d6056f3a771" (UID: "9a1891c1-4602-4f3f-b195-5d6056f3a771"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:07.922710 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.922681    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-dshm" (OuterVolumeSpecName: "dshm") pod "9a1891c1-4602-4f3f-b195-5d6056f3a771" (UID: "9a1891c1-4602-4f3f-b195-5d6056f3a771"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:07.922858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.922812    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a1891c1-4602-4f3f-b195-5d6056f3a771-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9a1891c1-4602-4f3f-b195-5d6056f3a771" (UID: "9a1891c1-4602-4f3f-b195-5d6056f3a771"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:07.922951 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:07.922931    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a1891c1-4602-4f3f-b195-5d6056f3a771-kube-api-access-jwzsf" (OuterVolumeSpecName: "kube-api-access-jwzsf") pod "9a1891c1-4602-4f3f-b195-5d6056f3a771" (UID: "9a1891c1-4602-4f3f-b195-5d6056f3a771"). InnerVolumeSpecName "kube-api-access-jwzsf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:08.021424 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.021401    2577 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9a1891c1-4602-4f3f-b195-5d6056f3a771-tls-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:57:08.021424 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.021424    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwzsf\" (UniqueName: \"kubernetes.io/projected/9a1891c1-4602-4f3f-b195-5d6056f3a771-kube-api-access-jwzsf\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:57:08.021558 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.021433    2577 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-dshm\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:57:08.021558 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.021442    2577 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-model-cache\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:57:08.021558 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.021451    2577 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9a1891c1-4602-4f3f-b195-5d6056f3a771-tmp-dir\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:57:08.134047 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.134021    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"]
Sep 10 15:57:08.136424 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.136399    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-6f8c71/rate-test-model-6f8c71-kserve-77589548bc-z6fpp"]
Sep 10 15:57:08.908430 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:08.908401    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a1891c1-4602-4f3f-b195-5d6056f3a771" path="/var/lib/kubelet/pods/9a1891c1-4602-4f3f-b195-5d6056f3a771/volumes"
Sep 10 15:57:10.773445 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:10.773407    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:57:26.864550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.864517    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"]
Sep 10 15:57:26.865169 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.865144    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e2307c6-a64d-406a-a3b5-e6ce7992e96f" containerName="maas-api"
Sep 10 15:57:26.865169 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.865165    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e2307c6-a64d-406a-a3b5-e6ce7992e96f" containerName="maas-api"
Sep 10 15:57:26.865357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.865179    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a1891c1-4602-4f3f-b195-5d6056f3a771" containerName="main"
Sep 10 15:57:26.865357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.865189    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a1891c1-4602-4f3f-b195-5d6056f3a771" containerName="main"
Sep 10 15:57:26.865357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.865291    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a1891c1-4602-4f3f-b195-5d6056f3a771" containerName="main"
Sep 10 15:57:26.865357 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.865304    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e2307c6-a64d-406a-a3b5-e6ce7992e96f" containerName="maas-api"
Sep 10 15:57:26.870114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.870092    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:26.872459 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.872436    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-f52559-serving-cert\""
Sep 10 15:57:26.880266 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.880247    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"]
Sep 10 15:57:26.988555 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.988520    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cbacef-d787-4249-8dda-c0b679a1387e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:26.988717 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.988570    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd9xf\" (UniqueName: \"kubernetes.io/projected/69cbacef-d787-4249-8dda-c0b679a1387e-kube-api-access-gd9xf\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:26.988717 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.988643    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:26.988879 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:26.988737    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.089839 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.089806    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.090007 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.089864    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cbacef-d787-4249-8dda-c0b679a1387e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.090007 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.089892    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd9xf\" (UniqueName: \"kubernetes.io/projected/69cbacef-d787-4249-8dda-c0b679a1387e-kube-api-access-gd9xf\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.090123 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.090054    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.090192 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:57:27.090175    2577 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-serving-cert: secret "maas-api-e2e-ipp-b-f52559-serving-cert" not found
Sep 10 15:57:27.090255 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:57:27.090245    2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls podName:69cbacef-d787-4249-8dda-c0b679a1387e nodeName:}" failed. No retries permitted until 2026-09-10 15:57:27.590227065 +0000 UTC m=+1491.204353948 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls") pod "maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76" (UID: "69cbacef-d787-4249-8dda-c0b679a1387e") : secret "maas-api-e2e-ipp-b-f52559-serving-cert" not found
Sep 10 15:57:27.090782 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.090688    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cbacef-d787-4249-8dda-c0b679a1387e-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.092806 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.092780    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-metrics-certs\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.099672 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.099649    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd9xf\" (UniqueName: \"kubernetes.io/projected/69cbacef-d787-4249-8dda-c0b679a1387e-kube-api-access-gd9xf\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.594132 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.594099    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.596632 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.596600    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls\") pod \"maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:27.785365 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:27.785331    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:28.056696 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:28.056671    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"]
Sep 10 15:57:28.057690 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:57:28.057667    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69cbacef_d787_4249_8dda_c0b679a1387e.slice/crio-23305fea79af92f0565d863e9bc10b5b2f6ecf6f8f93341a3702e84aa4da6a28 WatchSource:0}: Error finding container 23305fea79af92f0565d863e9bc10b5b2f6ecf6f8f93341a3702e84aa4da6a28: Status 404 returned error can't find the container with id 23305fea79af92f0565d863e9bc10b5b2f6ecf6f8f93341a3702e84aa4da6a28
Sep 10 15:57:28.935814 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:28.935780    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76" event={"ID":"69cbacef-d787-4249-8dda-c0b679a1387e","Type":"ContainerStarted","Data":"0a8c5df58e82a3fe917aec3dc794b6b2354e7281e4ecb4c09f1deb1005b20bf5"}
Sep 10 15:57:28.935974 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:28.935827    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76" event={"ID":"69cbacef-d787-4249-8dda-c0b679a1387e","Type":"ContainerStarted","Data":"23305fea79af92f0565d863e9bc10b5b2f6ecf6f8f93341a3702e84aa4da6a28"}
Sep 10 15:57:28.935974 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:28.935929    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:57:28.953848 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:28.953796    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76" podStartSLOduration=2.515758379 podStartE2EDuration="2.95378338s" totalImagesPullingTime="438.025001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:28.951312132 +0000 UTC m=+1492.565439041" watchObservedRunningTime="2026-09-10 15:57:28.95378338 +0000 UTC m=+1492.567910281"
Sep 10 15:57:34.946569 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:57:34.946545    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 15:58:42.887572 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.887541    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"]
Sep 10 15:58:42.891525 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.891495    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:42.894361 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.894335    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-7388f5-dockercfg-qtpng\""
Sep 10 15:58:42.894705 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.894675    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-7388f5\""
Sep 10 15:58:42.909296 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.909275    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"]
Sep 10 15:58:42.921727 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.921697    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"]
Sep 10 15:58:42.925205 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.925184    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:42.937311 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:42.937291    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"]
Sep 10 15:58:43.032905 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.032876    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxmzr\" (UniqueName: \"kubernetes.io/projected/a051e53e-b2d1-4149-b86b-80aae2730920-kube-api-access-vxmzr\") pod \"payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:43.033073 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.032930    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96257992-ddec-4663-9021-1bea29385fc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:43.033073 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.032959    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxdkn\" (UniqueName: \"kubernetes.io/projected/96257992-ddec-4663-9021-1bea29385fc5-kube-api-access-xxdkn\") pod \"payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:43.033073 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.033039    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a051e53e-b2d1-4149-b86b-80aae2730920-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:43.133830 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.133796    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxmzr\" (UniqueName: \"kubernetes.io/projected/a051e53e-b2d1-4149-b86b-80aae2730920-kube-api-access-vxmzr\") pod \"payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:43.133999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.133849    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96257992-ddec-4663-9021-1bea29385fc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:43.133999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.133873    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxdkn\" (UniqueName: \"kubernetes.io/projected/96257992-ddec-4663-9021-1bea29385fc5-kube-api-access-xxdkn\") pod \"payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:43.133999 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.133896    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a051e53e-b2d1-4149-b86b-80aae2730920-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:43.134597 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.134577    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a051e53e-b2d1-4149-b86b-80aae2730920-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:43.134634 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.134595    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96257992-ddec-4663-9021-1bea29385fc5-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:43.142013 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.141953    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxmzr\" (UniqueName: \"kubernetes.io/projected/a051e53e-b2d1-4149-b86b-80aae2730920-kube-api-access-vxmzr\") pod \"payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:43.142013 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.141990    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxdkn\" (UniqueName: \"kubernetes.io/projected/96257992-ddec-4663-9021-1bea29385fc5-kube-api-access-xxdkn\") pod \"payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:43.203966 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.203942    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:43.235152 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.235115    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:43.478114 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.478089    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"]
Sep 10 15:58:43.480151 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:58:43.480123    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda051e53e_b2d1_4149_b86b_80aae2730920.slice/crio-5e95c054160555d3e5e6c13a25a5154b01c9d035669bf8161b757ea205c00b34 WatchSource:0}: Error finding container 5e95c054160555d3e5e6c13a25a5154b01c9d035669bf8161b757ea205c00b34: Status 404 returned error can't find the container with id 5e95c054160555d3e5e6c13a25a5154b01c9d035669bf8161b757ea205c00b34
Sep 10 15:58:43.520199 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:43.520178    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"]
Sep 10 15:58:43.521468 ip-10-0-139-11 kubenswrapper[2577]: W0910 15:58:43.521439    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96257992_ddec_4663_9021_1bea29385fc5.slice/crio-74cb08ae37ec81da96ed995ec923ba61fb1f555064a8d9cc22becac546323e5c WatchSource:0}: Error finding container 74cb08ae37ec81da96ed995ec923ba61fb1f555064a8d9cc22becac546323e5c: Status 404 returned error can't find the container with id 74cb08ae37ec81da96ed995ec923ba61fb1f555064a8d9cc22becac546323e5c
Sep 10 15:58:44.358680 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.358646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq" event={"ID":"96257992-ddec-4663-9021-1bea29385fc5","Type":"ContainerStarted","Data":"25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb"}
Sep 10 15:58:44.359187 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.358700    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq" event={"ID":"96257992-ddec-4663-9021-1bea29385fc5","Type":"ContainerStarted","Data":"74cb08ae37ec81da96ed995ec923ba61fb1f555064a8d9cc22becac546323e5c"}
Sep 10 15:58:44.359187 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.358841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:44.360441 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.360411    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk" event={"ID":"a051e53e-b2d1-4149-b86b-80aae2730920","Type":"ContainerStarted","Data":"0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607"}
Sep 10 15:58:44.360557 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.360460    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:58:44.360557 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.360477    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk" event={"ID":"a051e53e-b2d1-4149-b86b-80aae2730920","Type":"ContainerStarted","Data":"5e95c054160555d3e5e6c13a25a5154b01c9d035669bf8161b757ea205c00b34"}
Sep 10 15:58:44.375935 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.375891    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq" podStartSLOduration=2.375879059 podStartE2EDuration="2.375879059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:44.375433503 +0000 UTC m=+1567.989560408" watchObservedRunningTime="2026-09-10 15:58:44.375879059 +0000 UTC m=+1567.990005963"
Sep 10 15:58:44.394095 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:44.394045    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk" podStartSLOduration=2.394027724 podStartE2EDuration="2.394027724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:44.390837443 +0000 UTC m=+1568.004964349" watchObservedRunningTime="2026-09-10 15:58:44.394027724 +0000 UTC m=+1568.008154629"
Sep 10 15:58:55.367876 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:55.367841    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:58:55.368322 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:58:55.367936    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:59:12.922988 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:12.922957    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"]
Sep 10 15:59:12.923522 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:12.923181    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq" podUID="96257992-ddec-4663-9021-1bea29385fc5" containerName="payload-processing" containerID="cri-o://25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb" gracePeriod=30
Sep 10 15:59:12.934213 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:12.933921    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"]
Sep 10 15:59:12.934362 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:12.934181    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk" podUID="a051e53e-b2d1-4149-b86b-80aae2730920" containerName="payload-pre-processing" containerID="cri-o://0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607" gracePeriod=30
Sep 10 15:59:13.206519 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.206493    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:59:13.210547 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.210524    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:59:13.398229 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.398190    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxmzr\" (UniqueName: \"kubernetes.io/projected/a051e53e-b2d1-4149-b86b-80aae2730920-kube-api-access-vxmzr\") pod \"a051e53e-b2d1-4149-b86b-80aae2730920\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") "
Sep 10 15:59:13.398412 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.398289    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a051e53e-b2d1-4149-b86b-80aae2730920-plugins-config-volume\") pod \"a051e53e-b2d1-4149-b86b-80aae2730920\" (UID: \"a051e53e-b2d1-4149-b86b-80aae2730920\") "
Sep 10 15:59:13.398412 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.398339    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96257992-ddec-4663-9021-1bea29385fc5-plugins-config-volume\") pod \"96257992-ddec-4663-9021-1bea29385fc5\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") "
Sep 10 15:59:13.398412 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.398377    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxdkn\" (UniqueName: \"kubernetes.io/projected/96257992-ddec-4663-9021-1bea29385fc5-kube-api-access-xxdkn\") pod \"96257992-ddec-4663-9021-1bea29385fc5\" (UID: \"96257992-ddec-4663-9021-1bea29385fc5\") "
Sep 10 15:59:13.398733 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.398707    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96257992-ddec-4663-9021-1bea29385fc5-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96257992-ddec-4663-9021-1bea29385fc5" (UID: "96257992-ddec-4663-9021-1bea29385fc5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:13.398839 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.398706    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a051e53e-b2d1-4149-b86b-80aae2730920-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a051e53e-b2d1-4149-b86b-80aae2730920" (UID: "a051e53e-b2d1-4149-b86b-80aae2730920"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:13.400585 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.400561    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a051e53e-b2d1-4149-b86b-80aae2730920-kube-api-access-vxmzr" (OuterVolumeSpecName: "kube-api-access-vxmzr") pod "a051e53e-b2d1-4149-b86b-80aae2730920" (UID: "a051e53e-b2d1-4149-b86b-80aae2730920"). InnerVolumeSpecName "kube-api-access-vxmzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:13.400585 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.400576    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96257992-ddec-4663-9021-1bea29385fc5-kube-api-access-xxdkn" (OuterVolumeSpecName: "kube-api-access-xxdkn") pod "96257992-ddec-4663-9021-1bea29385fc5" (UID: "96257992-ddec-4663-9021-1bea29385fc5"). InnerVolumeSpecName "kube-api-access-xxdkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:13.499086 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.499006    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a051e53e-b2d1-4149-b86b-80aae2730920-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:13.499086 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.499033    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96257992-ddec-4663-9021-1bea29385fc5-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:13.499086 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.499042    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxdkn\" (UniqueName: \"kubernetes.io/projected/96257992-ddec-4663-9021-1bea29385fc5-kube-api-access-xxdkn\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:13.499086 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.499051    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxmzr\" (UniqueName: \"kubernetes.io/projected/a051e53e-b2d1-4149-b86b-80aae2730920-kube-api-access-vxmzr\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:13.521293 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.521265    2577 generic.go:356] "Generic (PLEG): container finished" podID="96257992-ddec-4663-9021-1bea29385fc5" containerID="25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb" exitCode=0
Sep 10 15:59:13.521436 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.521331    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"
Sep 10 15:59:13.521436 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.521374    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq" event={"ID":"96257992-ddec-4663-9021-1bea29385fc5","Type":"ContainerDied","Data":"25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb"}
Sep 10 15:59:13.521547 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.521453    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq" event={"ID":"96257992-ddec-4663-9021-1bea29385fc5","Type":"ContainerDied","Data":"74cb08ae37ec81da96ed995ec923ba61fb1f555064a8d9cc22becac546323e5c"}
Sep 10 15:59:13.521547 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.521488    2577 scope.go:117] "RemoveContainer" containerID="25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb"
Sep 10 15:59:13.522822 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.522798    2577 generic.go:356] "Generic (PLEG): container finished" podID="a051e53e-b2d1-4149-b86b-80aae2730920" containerID="0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607" exitCode=0
Sep 10 15:59:13.522956 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.522862    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"
Sep 10 15:59:13.522956 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.522882    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk" event={"ID":"a051e53e-b2d1-4149-b86b-80aae2730920","Type":"ContainerDied","Data":"0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607"}
Sep 10 15:59:13.522956 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.522930    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk" event={"ID":"a051e53e-b2d1-4149-b86b-80aae2730920","Type":"ContainerDied","Data":"5e95c054160555d3e5e6c13a25a5154b01c9d035669bf8161b757ea205c00b34"}
Sep 10 15:59:13.530674 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.530657    2577 scope.go:117] "RemoveContainer" containerID="25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb"
Sep 10 15:59:13.530989 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:59:13.530967    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb\": container with ID starting with 25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb not found: ID does not exist" containerID="25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb"
Sep 10 15:59:13.531161 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.530997    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb"} err="failed to get container status \"25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb\": rpc error: code = NotFound desc = could not find container \"25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb\": container with ID starting with 25c1ee95eb426b7f5a7da4e83804c27c82365c856cab04566d89377279eacedb not found: ID does not exist"
Sep 10 15:59:13.531161 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.531013    2577 scope.go:117] "RemoveContainer" containerID="0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607"
Sep 10 15:59:13.539242 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.539218    2577 scope.go:117] "RemoveContainer" containerID="0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607"
Sep 10 15:59:13.539492 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:59:13.539474    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607\": container with ID starting with 0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607 not found: ID does not exist" containerID="0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607"
Sep 10 15:59:13.539550 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.539501    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607"} err="failed to get container status \"0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607\": rpc error: code = NotFound desc = could not find container \"0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607\": container with ID starting with 0dc75fec0af8eb173557ac71c6c9e1516f69e3e3c3a81de9817a992c89823607 not found: ID does not exist"
Sep 10 15:59:13.546338 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.546315    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"]
Sep 10 15:59:13.548521 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.548499    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-7388f5-77865f745d-765pk"]
Sep 10 15:59:13.557224 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.557203    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"]
Sep 10 15:59:13.559138 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:13.559118    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-7388f5-7df79d74c7-9cvrq"]
Sep 10 15:59:14.908923 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:14.908888    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96257992-ddec-4663-9021-1bea29385fc5" path="/var/lib/kubelet/pods/96257992-ddec-4663-9021-1bea29385fc5/volumes"
Sep 10 15:59:14.909426 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:14.909387    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a051e53e-b2d1-4149-b86b-80aae2730920" path="/var/lib/kubelet/pods/a051e53e-b2d1-4149-b86b-80aae2730920/volumes"
Sep 10 15:59:38.536714 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.536683    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"]
Sep 10 15:59:38.537259 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.536964    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r" podUID="078262ba-701a-46a9-a951-d258d5fd6c37" containerName="maas-api" containerID="cri-o://47b50f5abf9cba23fdbfdc07a0712069f3259e861f5129e65b0c0dc8a5594364" gracePeriod=30
Sep 10 15:59:38.600955 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.600924    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"]
Sep 10 15:59:38.601177 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.601152    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds" podUID="bce838d5-ea35-4c54-97b1-32d106646e27" containerName="payload-pre-processing" containerID="cri-o://822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e" gracePeriod=30
Sep 10 15:59:38.671124 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.671096    2577 generic.go:356] "Generic (PLEG): container finished" podID="078262ba-701a-46a9-a951-d258d5fd6c37" containerID="47b50f5abf9cba23fdbfdc07a0712069f3259e861f5129e65b0c0dc8a5594364" exitCode=0
Sep 10 15:59:38.671273 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.671183    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r" event={"ID":"078262ba-701a-46a9-a951-d258d5fd6c37","Type":"ContainerDied","Data":"47b50f5abf9cba23fdbfdc07a0712069f3259e861f5129e65b0c0dc8a5594364"}
Sep 10 15:59:38.851489 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.851464    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:59:38.887121 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.887100    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:59:38.899466 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.899443    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls\") pod \"078262ba-701a-46a9-a951-d258d5fd6c37\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") "
Sep 10 15:59:38.899584 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.899562    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078262ba-701a-46a9-a951-d258d5fd6c37-openshift-service-ca\") pod \"078262ba-701a-46a9-a951-d258d5fd6c37\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") "
Sep 10 15:59:38.899642 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.899596    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-metrics-certs\") pod \"078262ba-701a-46a9-a951-d258d5fd6c37\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") "
Sep 10 15:59:38.899681 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.899654    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lf5ng\" (UniqueName: \"kubernetes.io/projected/078262ba-701a-46a9-a951-d258d5fd6c37-kube-api-access-lf5ng\") pod \"078262ba-701a-46a9-a951-d258d5fd6c37\" (UID: \"078262ba-701a-46a9-a951-d258d5fd6c37\") "
Sep 10 15:59:38.900006 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.899972    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/078262ba-701a-46a9-a951-d258d5fd6c37-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "078262ba-701a-46a9-a951-d258d5fd6c37" (UID: "078262ba-701a-46a9-a951-d258d5fd6c37"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:38.901858 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.901834    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "078262ba-701a-46a9-a951-d258d5fd6c37" (UID: "078262ba-701a-46a9-a951-d258d5fd6c37"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:38.901965 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.901874    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "078262ba-701a-46a9-a951-d258d5fd6c37" (UID: "078262ba-701a-46a9-a951-d258d5fd6c37"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:38.902470 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:38.902267    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/078262ba-701a-46a9-a951-d258d5fd6c37-kube-api-access-lf5ng" (OuterVolumeSpecName: "kube-api-access-lf5ng") pod "078262ba-701a-46a9-a951-d258d5fd6c37" (UID: "078262ba-701a-46a9-a951-d258d5fd6c37"). InnerVolumeSpecName "kube-api-access-lf5ng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:39.000426 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.000399    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce838d5-ea35-4c54-97b1-32d106646e27-plugins-config-volume\") pod \"bce838d5-ea35-4c54-97b1-32d106646e27\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") "
Sep 10 15:59:39.000573 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.000450    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgfcr\" (UniqueName: \"kubernetes.io/projected/bce838d5-ea35-4c54-97b1-32d106646e27-kube-api-access-rgfcr\") pod \"bce838d5-ea35-4c54-97b1-32d106646e27\" (UID: \"bce838d5-ea35-4c54-97b1-32d106646e27\") "
Sep 10 15:59:39.000639 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.000626    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lf5ng\" (UniqueName: \"kubernetes.io/projected/078262ba-701a-46a9-a951-d258d5fd6c37-kube-api-access-lf5ng\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:39.000688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.000641    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:39.000688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.000652    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/078262ba-701a-46a9-a951-d258d5fd6c37-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:39.000688 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.000661    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/078262ba-701a-46a9-a951-d258d5fd6c37-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:39.000841 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.000816    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bce838d5-ea35-4c54-97b1-32d106646e27-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bce838d5-ea35-4c54-97b1-32d106646e27" (UID: "bce838d5-ea35-4c54-97b1-32d106646e27"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:39.002628 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.002607    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bce838d5-ea35-4c54-97b1-32d106646e27-kube-api-access-rgfcr" (OuterVolumeSpecName: "kube-api-access-rgfcr") pod "bce838d5-ea35-4c54-97b1-32d106646e27" (UID: "bce838d5-ea35-4c54-97b1-32d106646e27"). InnerVolumeSpecName "kube-api-access-rgfcr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:39.102082 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.102051    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bce838d5-ea35-4c54-97b1-32d106646e27-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:39.102222 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.102093    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgfcr\" (UniqueName: \"kubernetes.io/projected/bce838d5-ea35-4c54-97b1-32d106646e27-kube-api-access-rgfcr\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 15:59:39.677528 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.677491    2577 generic.go:356] "Generic (PLEG): container finished" podID="bce838d5-ea35-4c54-97b1-32d106646e27" containerID="822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e" exitCode=0
Sep 10 15:59:39.678038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.677539    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"
Sep 10 15:59:39.678038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.677571    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds" event={"ID":"bce838d5-ea35-4c54-97b1-32d106646e27","Type":"ContainerDied","Data":"822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e"}
Sep 10 15:59:39.678038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.677633    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds" event={"ID":"bce838d5-ea35-4c54-97b1-32d106646e27","Type":"ContainerDied","Data":"970f9f640af8826b89761c7f83e210b116ce652e3b4519c0bb873707772c9a76"}
Sep 10 15:59:39.678038 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.677655    2577 scope.go:117] "RemoveContainer" containerID="822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e"
Sep 10 15:59:39.679188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.679012    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r" event={"ID":"078262ba-701a-46a9-a951-d258d5fd6c37","Type":"ContainerDied","Data":"cd4a1b2109f2e2fa6ae1e1eead973006aa039ab0c5fc29b5f54ed3c8116454ae"}
Sep 10 15:59:39.679188 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.679056    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"
Sep 10 15:59:39.687712 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.687698    2577 scope.go:117] "RemoveContainer" containerID="822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e"
Sep 10 15:59:39.687994 ip-10-0-139-11 kubenswrapper[2577]: E0910 15:59:39.687973    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e\": container with ID starting with 822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e not found: ID does not exist" containerID="822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e"
Sep 10 15:59:39.688098 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.687999    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e"} err="failed to get container status \"822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e\": rpc error: code = NotFound desc = could not find container \"822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e\": container with ID starting with 822b4a1ee4490351be986dbe234fc8ec3ae9ca17cb2311259bce5a3bcb10633e not found: ID does not exist"
Sep 10 15:59:39.688098 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.688016    2577 scope.go:117] "RemoveContainer" containerID="47b50f5abf9cba23fdbfdc07a0712069f3259e861f5129e65b0c0dc8a5594364"
Sep 10 15:59:39.700612 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.700588    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"]
Sep 10 15:59:39.707090 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.707071    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a85a91-78fc4ddd94-tggds"]
Sep 10 15:59:39.716467 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.716449    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"]
Sep 10 15:59:39.719972 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:39.719954    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-a85a91-b6cd8c4f5-nnj7r"]
Sep 10 15:59:40.908301 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:40.908268    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="078262ba-701a-46a9-a951-d258d5fd6c37" path="/var/lib/kubelet/pods/078262ba-701a-46a9-a951-d258d5fd6c37/volumes"
Sep 10 15:59:40.908664 ip-10-0-139-11 kubenswrapper[2577]: I0910 15:59:40.908621    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bce838d5-ea35-4c54-97b1-32d106646e27" path="/var/lib/kubelet/pods/bce838d5-ea35-4c54-97b1-32d106646e27/volumes"
Sep 10 16:00:37.878993 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:37.878954    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"]
Sep 10 16:00:37.879517 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:37.879179    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76" podUID="69cbacef-d787-4249-8dda-c0b679a1387e" containerName="maas-api" containerID="cri-o://0a8c5df58e82a3fe917aec3dc794b6b2354e7281e4ecb4c09f1deb1005b20bf5" gracePeriod=30
Sep 10 16:00:38.004232 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.004198    2577 generic.go:356] "Generic (PLEG): container finished" podID="69cbacef-d787-4249-8dda-c0b679a1387e" containerID="0a8c5df58e82a3fe917aec3dc794b6b2354e7281e4ecb4c09f1deb1005b20bf5" exitCode=0
Sep 10 16:00:38.004394 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.004272    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76" event={"ID":"69cbacef-d787-4249-8dda-c0b679a1387e","Type":"ContainerDied","Data":"0a8c5df58e82a3fe917aec3dc794b6b2354e7281e4ecb4c09f1deb1005b20bf5"}
Sep 10 16:00:38.135905 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.135853    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 16:00:38.217793 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.217761    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gd9xf\" (UniqueName: \"kubernetes.io/projected/69cbacef-d787-4249-8dda-c0b679a1387e-kube-api-access-gd9xf\") pod \"69cbacef-d787-4249-8dda-c0b679a1387e\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") "
Sep 10 16:00:38.217961 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.217803    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-metrics-certs\") pod \"69cbacef-d787-4249-8dda-c0b679a1387e\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") "
Sep 10 16:00:38.217961 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.217844    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls\") pod \"69cbacef-d787-4249-8dda-c0b679a1387e\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") "
Sep 10 16:00:38.217961 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.217876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cbacef-d787-4249-8dda-c0b679a1387e-openshift-service-ca\") pod \"69cbacef-d787-4249-8dda-c0b679a1387e\" (UID: \"69cbacef-d787-4249-8dda-c0b679a1387e\") "
Sep 10 16:00:38.218305 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.218275    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69cbacef-d787-4249-8dda-c0b679a1387e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69cbacef-d787-4249-8dda-c0b679a1387e" (UID: "69cbacef-d787-4249-8dda-c0b679a1387e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:38.220357 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.220330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69cbacef-d787-4249-8dda-c0b679a1387e" (UID: "69cbacef-d787-4249-8dda-c0b679a1387e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:00:38.220477 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.220410    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69cbacef-d787-4249-8dda-c0b679a1387e-kube-api-access-gd9xf" (OuterVolumeSpecName: "kube-api-access-gd9xf") pod "69cbacef-d787-4249-8dda-c0b679a1387e" (UID: "69cbacef-d787-4249-8dda-c0b679a1387e"). InnerVolumeSpecName "kube-api-access-gd9xf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:38.220602 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.220581    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "69cbacef-d787-4249-8dda-c0b679a1387e" (UID: "69cbacef-d787-4249-8dda-c0b679a1387e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:00:38.318787 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.318738    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:38.318787 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.318784    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69cbacef-d787-4249-8dda-c0b679a1387e-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:38.318787 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.318793    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd9xf\" (UniqueName: \"kubernetes.io/projected/69cbacef-d787-4249-8dda-c0b679a1387e-kube-api-access-gd9xf\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:38.319004 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:38.318801    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/69cbacef-d787-4249-8dda-c0b679a1387e-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:39.011230 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:39.011198    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"
Sep 10 16:00:39.011675 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:39.011229    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76" event={"ID":"69cbacef-d787-4249-8dda-c0b679a1387e","Type":"ContainerDied","Data":"23305fea79af92f0565d863e9bc10b5b2f6ecf6f8f93341a3702e84aa4da6a28"}
Sep 10 16:00:39.011675 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:39.011296    2577 scope.go:117] "RemoveContainer" containerID="0a8c5df58e82a3fe917aec3dc794b6b2354e7281e4ecb4c09f1deb1005b20bf5"
Sep 10 16:00:39.028642 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:39.028615    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"]
Sep 10 16:00:39.034182 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:39.034159    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-f52559-846c48c84c-5hg76"]
Sep 10 16:00:40.907692 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:40.907662    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69cbacef-d787-4249-8dda-c0b679a1387e" path="/var/lib/kubelet/pods/69cbacef-d787-4249-8dda-c0b679a1387e/volumes"
Sep 10 16:00:55.192272 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.192237    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"]
Sep 10 16:00:55.192879 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.192504    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq" podUID="bdeb1ef0-1051-4025-8ba3-468720297c54" containerName="maas-api" containerID="cri-o://4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc" gracePeriod=30
Sep 10 16:00:55.242609 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.242571    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"]
Sep 10 16:00:55.242913 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.242865    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9" podUID="9aee3627-d9b2-4307-be76-d3f1e73d3130" containerName="payload-processing" containerID="cri-o://2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882" gracePeriod=30
Sep 10 16:00:55.497949 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.497921    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 16:00:55.525523 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.525494    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 16:00:55.564813 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.564785    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-metrics-certs\") pod \"bdeb1ef0-1051-4025-8ba3-468720297c54\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") "
Sep 10 16:00:55.564995 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.564876    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls\") pod \"bdeb1ef0-1051-4025-8ba3-468720297c54\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") "
Sep 10 16:00:55.564995 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.564902    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sd7ls\" (UniqueName: \"kubernetes.io/projected/bdeb1ef0-1051-4025-8ba3-468720297c54-kube-api-access-sd7ls\") pod \"bdeb1ef0-1051-4025-8ba3-468720297c54\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") "
Sep 10 16:00:55.564995 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.564924    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeb1ef0-1051-4025-8ba3-468720297c54-openshift-service-ca\") pod \"bdeb1ef0-1051-4025-8ba3-468720297c54\" (UID: \"bdeb1ef0-1051-4025-8ba3-468720297c54\") "
Sep 10 16:00:55.565387 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.565350    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdeb1ef0-1051-4025-8ba3-468720297c54-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "bdeb1ef0-1051-4025-8ba3-468720297c54" (UID: "bdeb1ef0-1051-4025-8ba3-468720297c54"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:55.567072 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.567040    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "bdeb1ef0-1051-4025-8ba3-468720297c54" (UID: "bdeb1ef0-1051-4025-8ba3-468720297c54"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:00:55.567214 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.567147    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "bdeb1ef0-1051-4025-8ba3-468720297c54" (UID: "bdeb1ef0-1051-4025-8ba3-468720297c54"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:00:55.567214 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.567166    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdeb1ef0-1051-4025-8ba3-468720297c54-kube-api-access-sd7ls" (OuterVolumeSpecName: "kube-api-access-sd7ls") pod "bdeb1ef0-1051-4025-8ba3-468720297c54" (UID: "bdeb1ef0-1051-4025-8ba3-468720297c54"). InnerVolumeSpecName "kube-api-access-sd7ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:55.665572 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.665538    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8cr7k\" (UniqueName: \"kubernetes.io/projected/9aee3627-d9b2-4307-be76-d3f1e73d3130-kube-api-access-8cr7k\") pod \"9aee3627-d9b2-4307-be76-d3f1e73d3130\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") "
Sep 10 16:00:55.665779 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.665649    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aee3627-d9b2-4307-be76-d3f1e73d3130-plugins-config-volume\") pod \"9aee3627-d9b2-4307-be76-d3f1e73d3130\" (UID: \"9aee3627-d9b2-4307-be76-d3f1e73d3130\") "
Sep 10 16:00:55.665838 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.665825    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:55.665884 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.665838    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd7ls\" (UniqueName: \"kubernetes.io/projected/bdeb1ef0-1051-4025-8ba3-468720297c54-kube-api-access-sd7ls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:55.665884 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.665847    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdeb1ef0-1051-4025-8ba3-468720297c54-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:55.665884 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.665856    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bdeb1ef0-1051-4025-8ba3-468720297c54-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:55.666047 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.666024    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9aee3627-d9b2-4307-be76-d3f1e73d3130-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9aee3627-d9b2-4307-be76-d3f1e73d3130" (UID: "9aee3627-d9b2-4307-be76-d3f1e73d3130"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:55.667765 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.667731    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9aee3627-d9b2-4307-be76-d3f1e73d3130-kube-api-access-8cr7k" (OuterVolumeSpecName: "kube-api-access-8cr7k") pod "9aee3627-d9b2-4307-be76-d3f1e73d3130" (UID: "9aee3627-d9b2-4307-be76-d3f1e73d3130"). InnerVolumeSpecName "kube-api-access-8cr7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:55.766942 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.766868    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8cr7k\" (UniqueName: \"kubernetes.io/projected/9aee3627-d9b2-4307-be76-d3f1e73d3130-kube-api-access-8cr7k\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:55.766942 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:55.766892    2577 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9aee3627-d9b2-4307-be76-d3f1e73d3130-plugins-config-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:00:56.103595 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.103557    2577 generic.go:356] "Generic (PLEG): container finished" podID="9aee3627-d9b2-4307-be76-d3f1e73d3130" containerID="2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882" exitCode=0
Sep 10 16:00:56.103816 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.103614    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"
Sep 10 16:00:56.103816 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.103646    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9" event={"ID":"9aee3627-d9b2-4307-be76-d3f1e73d3130","Type":"ContainerDied","Data":"2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882"}
Sep 10 16:00:56.103816 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.103698    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9" event={"ID":"9aee3627-d9b2-4307-be76-d3f1e73d3130","Type":"ContainerDied","Data":"fed6d4d978747cb4d4fd9af67b8f99b7a6873f772b06dbcc249670cc5aab9585"}
Sep 10 16:00:56.103816 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.103722    2577 scope.go:117] "RemoveContainer" containerID="2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882"
Sep 10 16:00:56.105171 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.105145    2577 generic.go:356] "Generic (PLEG): container finished" podID="bdeb1ef0-1051-4025-8ba3-468720297c54" containerID="4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc" exitCode=0
Sep 10 16:00:56.105286 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.105193    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"
Sep 10 16:00:56.105286 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.105213    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq" event={"ID":"bdeb1ef0-1051-4025-8ba3-468720297c54","Type":"ContainerDied","Data":"4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc"}
Sep 10 16:00:56.105286 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.105252    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq" event={"ID":"bdeb1ef0-1051-4025-8ba3-468720297c54","Type":"ContainerDied","Data":"cb03d73be26bb8fc151c596060cb8dd7eff378326026a3cb3e383e924ceb6918"}
Sep 10 16:00:56.122492 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.122470    2577 scope.go:117] "RemoveContainer" containerID="2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882"
Sep 10 16:00:56.122809 ip-10-0-139-11 kubenswrapper[2577]: E0910 16:00:56.122779    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882\": container with ID starting with 2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882 not found: ID does not exist" containerID="2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882"
Sep 10 16:00:56.122902 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.122820    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882"} err="failed to get container status \"2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882\": rpc error: code = NotFound desc = could not find container \"2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882\": container with ID starting with 2a89d0afc971feb4b4376f84793360c4a5e4103646cc305cf85a2fce76c31882 not found: ID does not exist"
Sep 10 16:00:56.122902 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.122859    2577 scope.go:117] "RemoveContainer" containerID="4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc"
Sep 10 16:00:56.130930 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.130910    2577 scope.go:117] "RemoveContainer" containerID="4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc"
Sep 10 16:00:56.131321 ip-10-0-139-11 kubenswrapper[2577]: E0910 16:00:56.131283    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc\": container with ID starting with 4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc not found: ID does not exist" containerID="4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc"
Sep 10 16:00:56.131412 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.131329    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc"} err="failed to get container status \"4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc\": rpc error: code = NotFound desc = could not find container \"4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc\": container with ID starting with 4804d262f623a60c1c951e8110f9919a7fcdf57a3636ee8903e0a8e74938e8bc not found: ID does not exist"
Sep 10 16:00:56.133295 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.133275    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"]
Sep 10 16:00:56.139339 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.139319    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-6f8c71-5cf65848c4-5f2tq"]
Sep 10 16:00:56.148696 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.148677    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"]
Sep 10 16:00:56.151902 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.151878    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-6f8c71-7978755cdf-l5tt9"]
Sep 10 16:00:56.912603 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.912574    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9aee3627-d9b2-4307-be76-d3f1e73d3130" path="/var/lib/kubelet/pods/9aee3627-d9b2-4307-be76-d3f1e73d3130/volumes"
Sep 10 16:00:56.912997 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:00:56.912983    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdeb1ef0-1051-4025-8ba3-468720297c54" path="/var/lib/kubelet/pods/bdeb1ef0-1051-4025-8ba3-468720297c54/volumes"
Sep 10 16:01:12.843069 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:12.842868    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"]
Sep 10 16:01:12.843977 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:12.843930    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk" podUID="13c276c6-b209-4f24-84e4-ccb1414ed004" containerName="maas-api" containerID="cri-o://325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694" gracePeriod=30
Sep 10 16:01:13.094436 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.094375    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 16:01:13.199585 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.199531    2577 generic.go:356] "Generic (PLEG): container finished" podID="13c276c6-b209-4f24-84e4-ccb1414ed004" containerID="325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694" exitCode=0
Sep 10 16:01:13.199585 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.199585    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk" event={"ID":"13c276c6-b209-4f24-84e4-ccb1414ed004","Type":"ContainerDied","Data":"325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694"}
Sep 10 16:01:13.199859 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.199620    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"
Sep 10 16:01:13.199859 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.199680    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk" event={"ID":"13c276c6-b209-4f24-84e4-ccb1414ed004","Type":"ContainerDied","Data":"918cb4744c79d19060600ddc996d00ad93d9dfc9560933c9d66ebb27439f0c9e"}
Sep 10 16:01:13.199859 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.199713    2577 scope.go:117] "RemoveContainer" containerID="325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694"
Sep 10 16:01:13.209913 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.209882    2577 scope.go:117] "RemoveContainer" containerID="325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694"
Sep 10 16:01:13.210200 ip-10-0-139-11 kubenswrapper[2577]: E0910 16:01:13.210181    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694\": container with ID starting with 325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694 not found: ID does not exist" containerID="325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694"
Sep 10 16:01:13.210275 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.210212    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694"} err="failed to get container status \"325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694\": rpc error: code = NotFound desc = could not find container \"325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694\": container with ID starting with 325cd033309f30de2b16808a1aa98077e50714915620d41cf51fc2dcc2c82694 not found: ID does not exist"
Sep 10 16:01:13.218007 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.217988    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls\") pod \"13c276c6-b209-4f24-84e4-ccb1414ed004\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") "
Sep 10 16:01:13.218111 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.218036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drxvp\" (UniqueName: \"kubernetes.io/projected/13c276c6-b209-4f24-84e4-ccb1414ed004-kube-api-access-drxvp\") pod \"13c276c6-b209-4f24-84e4-ccb1414ed004\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") "
Sep 10 16:01:13.218111 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.218068    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-metrics-certs\") pod \"13c276c6-b209-4f24-84e4-ccb1414ed004\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") "
Sep 10 16:01:13.218111 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.218097    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c276c6-b209-4f24-84e4-ccb1414ed004-openshift-service-ca\") pod \"13c276c6-b209-4f24-84e4-ccb1414ed004\" (UID: \"13c276c6-b209-4f24-84e4-ccb1414ed004\") "
Sep 10 16:01:13.218540 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.218515    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13c276c6-b209-4f24-84e4-ccb1414ed004-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "13c276c6-b209-4f24-84e4-ccb1414ed004" (UID: "13c276c6-b209-4f24-84e4-ccb1414ed004"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:01:13.220046 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.220025    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "13c276c6-b209-4f24-84e4-ccb1414ed004" (UID: "13c276c6-b209-4f24-84e4-ccb1414ed004"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:01:13.220259 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.220230    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "13c276c6-b209-4f24-84e4-ccb1414ed004" (UID: "13c276c6-b209-4f24-84e4-ccb1414ed004"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:01:13.220613 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.220597    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13c276c6-b209-4f24-84e4-ccb1414ed004-kube-api-access-drxvp" (OuterVolumeSpecName: "kube-api-access-drxvp") pod "13c276c6-b209-4f24-84e4-ccb1414ed004" (UID: "13c276c6-b209-4f24-84e4-ccb1414ed004"). InnerVolumeSpecName "kube-api-access-drxvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:13.319349 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.319319    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:01:13.319349 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.319345    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drxvp\" (UniqueName: \"kubernetes.io/projected/13c276c6-b209-4f24-84e4-ccb1414ed004-kube-api-access-drxvp\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:01:13.319349 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.319354    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/13c276c6-b209-4f24-84e4-ccb1414ed004-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:01:13.319538 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.319363    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/13c276c6-b209-4f24-84e4-ccb1414ed004-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:01:13.520736 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.520701    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"]
Sep 10 16:01:13.524275 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:13.524251    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-e412c4-5bc9b676b5-s7pjk"]
Sep 10 16:01:14.908018 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:14.907989    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13c276c6-b209-4f24-84e4-ccb1414ed004" path="/var/lib/kubelet/pods/13c276c6-b209-4f24-84e4-ccb1414ed004/volumes"
Sep 10 16:01:37.196332 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:01:37.196297    2577 scope.go:117] "RemoveContainer" containerID="31db1814fa2c4412c820a435a6a8876776f3f0de96b7337fd61765aa3d62523b"
Sep 10 16:02:24.244406 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244371    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-569c478d95-ffl4p"]
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244723    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdeb1ef0-1051-4025-8ba3-468720297c54" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244734    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdeb1ef0-1051-4025-8ba3-468720297c54" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244746    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9aee3627-d9b2-4307-be76-d3f1e73d3130" containerName="payload-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244769    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="9aee3627-d9b2-4307-be76-d3f1e73d3130" containerName="payload-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244786    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="078262ba-701a-46a9-a951-d258d5fd6c37" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244795    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="078262ba-701a-46a9-a951-d258d5fd6c37" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244803    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69cbacef-d787-4249-8dda-c0b679a1387e" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244809    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="69cbacef-d787-4249-8dda-c0b679a1387e" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244819    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13c276c6-b209-4f24-84e4-ccb1414ed004" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244826    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="13c276c6-b209-4f24-84e4-ccb1414ed004" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244835    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a051e53e-b2d1-4149-b86b-80aae2730920" containerName="payload-pre-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244841    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="a051e53e-b2d1-4149-b86b-80aae2730920" containerName="payload-pre-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244856    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96257992-ddec-4663-9021-1bea29385fc5" containerName="payload-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244865    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="96257992-ddec-4663-9021-1bea29385fc5" containerName="payload-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244878    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bce838d5-ea35-4c54-97b1-32d106646e27" containerName="payload-pre-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244887    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="bce838d5-ea35-4c54-97b1-32d106646e27" containerName="payload-pre-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244968    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="96257992-ddec-4663-9021-1bea29385fc5" containerName="payload-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244983    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bce838d5-ea35-4c54-97b1-32d106646e27" containerName="payload-pre-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.244993    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="9aee3627-d9b2-4307-be76-d3f1e73d3130" containerName="payload-processing"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.245002    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="078262ba-701a-46a9-a951-d258d5fd6c37" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.245012    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="13c276c6-b209-4f24-84e4-ccb1414ed004" containerName="maas-api"
Sep 10 16:02:24.245000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.245020    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdeb1ef0-1051-4025-8ba3-468720297c54" containerName="maas-api"
Sep 10 16:02:24.245822 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.245028    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="a051e53e-b2d1-4149-b86b-80aae2730920" containerName="payload-pre-processing"
Sep 10 16:02:24.245822 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.245037    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="69cbacef-d787-4249-8dda-c0b679a1387e" containerName="maas-api"
Sep 10 16:02:24.248080 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.248064    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.251732 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.251711    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 16:02:24.251860 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.251731    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 16:02:24.251860 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.251714    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w22h2\""
Sep 10 16:02:24.251860 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.251791    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 16:02:24.258018 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.257997    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569c478d95-ffl4p"]
Sep 10 16:02:24.314314 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.314285    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg79d\" (UniqueName: \"kubernetes.io/projected/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-kube-api-access-sg79d\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.314482 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.314326    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-metrics-certs\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.314482 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.314408    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-webhook-cert\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.415742 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.415706    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-webhook-cert\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.415913 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.415815    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg79d\" (UniqueName: \"kubernetes.io/projected/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-kube-api-access-sg79d\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.415913 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.415855    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-metrics-certs\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.418488 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.418462    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-webhook-cert\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.418608 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.418492    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-metrics-certs\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.424822 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.424793    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg79d\" (UniqueName: \"kubernetes.io/projected/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-kube-api-access-sg79d\") pod \"maas-controller-569c478d95-ffl4p\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") " pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.559931 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.559831    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:24.833331 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.833302    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569c478d95-ffl4p"]
Sep 10 16:02:24.834531 ip-10-0-139-11 kubenswrapper[2577]: W0910 16:02:24.834508    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod798c3ceb_ed8b_47bf_837e_5d7abd6fcc70.slice/crio-b9dbe28894bf8ca36040fff91d07a5cf12a5e9d586509f6ab4f561dd1a96c7d7 WatchSource:0}: Error finding container b9dbe28894bf8ca36040fff91d07a5cf12a5e9d586509f6ab4f561dd1a96c7d7: Status 404 returned error can't find the container with id b9dbe28894bf8ca36040fff91d07a5cf12a5e9d586509f6ab4f561dd1a96c7d7
Sep 10 16:02:24.835884 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:24.835863    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:02:25.581549 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:25.581519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-ffl4p" event={"ID":"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70","Type":"ContainerStarted","Data":"511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c"}
Sep 10 16:02:25.581956 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:25.581567    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-ffl4p" event={"ID":"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70","Type":"ContainerStarted","Data":"b9dbe28894bf8ca36040fff91d07a5cf12a5e9d586509f6ab4f561dd1a96c7d7"}
Sep 10 16:02:25.581956 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:25.581631    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:25.599573 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:25.599525    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-569c478d95-ffl4p" podStartSLOduration=0.978568033 podStartE2EDuration="1.599509526s" totalImagesPullingTime="620.941493ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:25.597378936 +0000 UTC m=+1789.211505840" watchObservedRunningTime="2026-09-10 16:02:25.599509526 +0000 UTC m=+1789.213636430"
Sep 10 16:02:36.590473 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:36.590445    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:45.922555 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:45.922522    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569c478d95-ffl4p"]
Sep 10 16:02:45.923085 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:45.922768    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-569c478d95-ffl4p" podUID="798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" containerName="manager" containerID="cri-o://511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c" gracePeriod=30
Sep 10 16:02:46.173462 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.173402    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:46.300987 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.300939    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-metrics-certs\") pod \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") "
Sep 10 16:02:46.301197 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.301036    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg79d\" (UniqueName: \"kubernetes.io/projected/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-kube-api-access-sg79d\") pod \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") "
Sep 10 16:02:46.301197 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.301071    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-webhook-cert\") pod \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\" (UID: \"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70\") "
Sep 10 16:02:46.303366 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.303330    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" (UID: "798c3ceb-ed8b-47bf-837e-5d7abd6fcc70"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:02:46.303366 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.303338    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" (UID: "798c3ceb-ed8b-47bf-837e-5d7abd6fcc70"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:02:46.303571 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.303339    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-kube-api-access-sg79d" (OuterVolumeSpecName: "kube-api-access-sg79d") pod "798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" (UID: "798c3ceb-ed8b-47bf-837e-5d7abd6fcc70"). InnerVolumeSpecName "kube-api-access-sg79d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:02:46.401922 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.401885    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:02:46.401922 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.401918    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg79d\" (UniqueName: \"kubernetes.io/projected/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-kube-api-access-sg79d\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:02:46.402125 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.401932    2577 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70-webhook-cert\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:02:46.692356 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.692317    2577 generic.go:356] "Generic (PLEG): container finished" podID="798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" containerID="511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c" exitCode=0
Sep 10 16:02:46.692532 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.692383    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-ffl4p"
Sep 10 16:02:46.692532 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.692402    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-ffl4p" event={"ID":"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70","Type":"ContainerDied","Data":"511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c"}
Sep 10 16:02:46.692532 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.692478    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-ffl4p" event={"ID":"798c3ceb-ed8b-47bf-837e-5d7abd6fcc70","Type":"ContainerDied","Data":"b9dbe28894bf8ca36040fff91d07a5cf12a5e9d586509f6ab4f561dd1a96c7d7"}
Sep 10 16:02:46.692532 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.692500    2577 scope.go:117] "RemoveContainer" containerID="511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c"
Sep 10 16:02:46.702805 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.702783    2577 scope.go:117] "RemoveContainer" containerID="511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c"
Sep 10 16:02:46.703104 ip-10-0-139-11 kubenswrapper[2577]: E0910 16:02:46.703081    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c\": container with ID starting with 511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c not found: ID does not exist" containerID="511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c"
Sep 10 16:02:46.703146 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.703116    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c"} err="failed to get container status \"511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c\": rpc error: code = NotFound desc = could not find container \"511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c\": container with ID starting with 511af1f6a6e9dd973ba169ea5c2eaf05697f39897d7ee966155ddfbdec4f6e1c not found: ID does not exist"
Sep 10 16:02:46.713568 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.713546    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-569c478d95-ffl4p"]
Sep 10 16:02:46.716946 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.716912    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-569c478d95-ffl4p"]
Sep 10 16:02:46.908009 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:46.907976    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" path="/var/lib/kubelet/pods/798c3ceb-ed8b-47bf-837e-5d7abd6fcc70/volumes"
Sep 10 16:02:47.513019 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.512985    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-569c478d95-qn5wh"]
Sep 10 16:02:47.513411 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.513327    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" containerName="manager"
Sep 10 16:02:47.513411 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.513338    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" containerName="manager"
Sep 10 16:02:47.513411 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.513407    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="798c3ceb-ed8b-47bf-837e-5d7abd6fcc70" containerName="manager"
Sep 10 16:02:47.517697 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.517676    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.520947 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.520928    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 10 16:02:47.521068 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.520948    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 10 16:02:47.521068 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.520930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-w22h2\""
Sep 10 16:02:47.521068 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.520930    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 10 16:02:47.525008 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.524980    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569c478d95-qn5wh"]
Sep 10 16:02:47.610903 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.610873    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf55be4-0755-416b-b4e3-a0633cd5d777-metrics-certs\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.611089 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.610936    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwz76\" (UniqueName: \"kubernetes.io/projected/8cf55be4-0755-416b-b4e3-a0633cd5d777-kube-api-access-qwz76\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.611089 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.610982    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cf55be4-0755-416b-b4e3-a0633cd5d777-webhook-cert\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.712200 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.712167    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwz76\" (UniqueName: \"kubernetes.io/projected/8cf55be4-0755-416b-b4e3-a0633cd5d777-kube-api-access-qwz76\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.712411 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.712219    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cf55be4-0755-416b-b4e3-a0633cd5d777-webhook-cert\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.712411 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.712278    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf55be4-0755-416b-b4e3-a0633cd5d777-metrics-certs\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.714926 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.714901    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cf55be4-0755-416b-b4e3-a0633cd5d777-metrics-certs\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.715040 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.715027    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8cf55be4-0755-416b-b4e3-a0633cd5d777-webhook-cert\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.720316 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.720293    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwz76\" (UniqueName: \"kubernetes.io/projected/8cf55be4-0755-416b-b4e3-a0633cd5d777-kube-api-access-qwz76\") pod \"maas-controller-569c478d95-qn5wh\" (UID: \"8cf55be4-0755-416b-b4e3-a0633cd5d777\") " pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:47.828899 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:47.828863    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:48.101840 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:48.101814    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-569c478d95-qn5wh"]
Sep 10 16:02:48.103815 ip-10-0-139-11 kubenswrapper[2577]: W0910 16:02:48.103789    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cf55be4_0755_416b_b4e3_a0633cd5d777.slice/crio-ff2454f7e2a7b8fcd79da148be731682867dd9d10b07175c990471dd582f62d5 WatchSource:0}: Error finding container ff2454f7e2a7b8fcd79da148be731682867dd9d10b07175c990471dd582f62d5: Status 404 returned error can't find the container with id ff2454f7e2a7b8fcd79da148be731682867dd9d10b07175c990471dd582f62d5
Sep 10 16:02:48.706535 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:48.706495    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-qn5wh" event={"ID":"8cf55be4-0755-416b-b4e3-a0633cd5d777","Type":"ContainerStarted","Data":"42b56ea5312bab80a6e72383b87ab6b24896d875552e7e61c9fc0fb0559a06cb"}
Sep 10 16:02:48.707032 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:48.706557    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-569c478d95-qn5wh" event={"ID":"8cf55be4-0755-416b-b4e3-a0633cd5d777","Type":"ContainerStarted","Data":"ff2454f7e2a7b8fcd79da148be731682867dd9d10b07175c990471dd582f62d5"}
Sep 10 16:02:48.707032 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:48.706588    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:02:48.724503 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:48.724456    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-569c478d95-qn5wh" podStartSLOduration=1.2875263270000001 podStartE2EDuration="1.72444056s" totalImagesPullingTime="436.914233ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:48.721565089 +0000 UTC m=+1812.335691993" watchObservedRunningTime="2026-09-10 16:02:48.72444056 +0000 UTC m=+1812.338567511"
Sep 10 16:02:59.714524 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:02:59.714495    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-569c478d95-qn5wh"
Sep 10 16:04:08.094435 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.094354    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"]
Sep 10 16:04:08.097036 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.094542    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz" podUID="d904b35c-d482-4e27-b14f-6031df978c0a" containerName="manager" containerID="cri-o://9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903" gracePeriod=10
Sep 10 16:04:08.443168 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.443142    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 16:04:08.605337 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.605302    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5p4pb\" (UniqueName: \"kubernetes.io/projected/d904b35c-d482-4e27-b14f-6031df978c0a-kube-api-access-5p4pb\") pod \"d904b35c-d482-4e27-b14f-6031df978c0a\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") "
Sep 10 16:04:08.605512 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.605341    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d904b35c-d482-4e27-b14f-6031df978c0a-extensions-socket-volume\") pod \"d904b35c-d482-4e27-b14f-6031df978c0a\" (UID: \"d904b35c-d482-4e27-b14f-6031df978c0a\") "
Sep 10 16:04:08.605799 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.605739    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d904b35c-d482-4e27-b14f-6031df978c0a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "d904b35c-d482-4e27-b14f-6031df978c0a" (UID: "d904b35c-d482-4e27-b14f-6031df978c0a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:04:08.607416 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.607395    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d904b35c-d482-4e27-b14f-6031df978c0a-kube-api-access-5p4pb" (OuterVolumeSpecName: "kube-api-access-5p4pb") pod "d904b35c-d482-4e27-b14f-6031df978c0a" (UID: "d904b35c-d482-4e27-b14f-6031df978c0a"). InnerVolumeSpecName "kube-api-access-5p4pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:08.706128 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.706067    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5p4pb\" (UniqueName: \"kubernetes.io/projected/d904b35c-d482-4e27-b14f-6031df978c0a-kube-api-access-5p4pb\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:04:08.706128 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:08.706088    2577 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/d904b35c-d482-4e27-b14f-6031df978c0a-extensions-socket-volume\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:04:09.132813 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.132782    2577 generic.go:356] "Generic (PLEG): container finished" podID="d904b35c-d482-4e27-b14f-6031df978c0a" containerID="9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903" exitCode=0
Sep 10 16:04:09.133234 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.132888    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"
Sep 10 16:04:09.133234 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.132925    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz" event={"ID":"d904b35c-d482-4e27-b14f-6031df978c0a","Type":"ContainerDied","Data":"9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903"}
Sep 10 16:04:09.133234 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.132980    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz" event={"ID":"d904b35c-d482-4e27-b14f-6031df978c0a","Type":"ContainerDied","Data":"c7eb52b7b4b63321a4a8ca277d13c96c6c65bed53c6d6af297601f4c7663f9d0"}
Sep 10 16:04:09.133234 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.133003    2577 scope.go:117] "RemoveContainer" containerID="9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903"
Sep 10 16:04:09.142607 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.142590    2577 scope.go:117] "RemoveContainer" containerID="9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903"
Sep 10 16:04:09.142889 ip-10-0-139-11 kubenswrapper[2577]: E0910 16:04:09.142871    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903\": container with ID starting with 9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903 not found: ID does not exist" containerID="9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903"
Sep 10 16:04:09.142967 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.142898    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903"} err="failed to get container status \"9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903\": rpc error: code = NotFound desc = could not find container \"9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903\": container with ID starting with 9e429274ae82cb3d368ce03ec5eadf497f799b03255e112552f2683134a60903 not found: ID does not exist"
Sep 10 16:04:09.149700 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.149675    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"]
Sep 10 16:04:09.152582 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:09.152563    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-ncllz"]
Sep 10 16:04:10.907116 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:04:10.907082    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d904b35c-d482-4e27-b14f-6031df978c0a" path="/var/lib/kubelet/pods/d904b35c-d482-4e27-b14f-6031df978c0a/volumes"
Sep 10 16:05:08.813263 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.813229    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"]
Sep 10 16:05:08.813729 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.813549    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d904b35c-d482-4e27-b14f-6031df978c0a" containerName="manager"
Sep 10 16:05:08.813729 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.813560    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="d904b35c-d482-4e27-b14f-6031df978c0a" containerName="manager"
Sep 10 16:05:08.813729 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.813632    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="d904b35c-d482-4e27-b14f-6031df978c0a" containerName="manager"
Sep 10 16:05:08.816455 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.816440    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:08.818802 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.818778    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-865x4\""
Sep 10 16:05:08.832048 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.832028    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"]
Sep 10 16:05:08.995622 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.995583    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf0fed77-0cfb-484c-8a47-9c73af7346fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6fprf\" (UID: \"bf0fed77-0cfb-484c-8a47-9c73af7346fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:08.995823 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:08.995632    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq72j\" (UniqueName: \"kubernetes.io/projected/bf0fed77-0cfb-484c-8a47-9c73af7346fc-kube-api-access-fq72j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6fprf\" (UID: \"bf0fed77-0cfb-484c-8a47-9c73af7346fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:09.096740 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:09.096708    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf0fed77-0cfb-484c-8a47-9c73af7346fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6fprf\" (UID: \"bf0fed77-0cfb-484c-8a47-9c73af7346fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:09.096914 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:09.096762    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq72j\" (UniqueName: \"kubernetes.io/projected/bf0fed77-0cfb-484c-8a47-9c73af7346fc-kube-api-access-fq72j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6fprf\" (UID: \"bf0fed77-0cfb-484c-8a47-9c73af7346fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:09.097091 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:09.097067    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf0fed77-0cfb-484c-8a47-9c73af7346fc-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6fprf\" (UID: \"bf0fed77-0cfb-484c-8a47-9c73af7346fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:09.104690 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:09.104661    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq72j\" (UniqueName: \"kubernetes.io/projected/bf0fed77-0cfb-484c-8a47-9c73af7346fc-kube-api-access-fq72j\") pod \"kuadrant-operator-controller-manager-5769f99d9f-6fprf\" (UID: \"bf0fed77-0cfb-484c-8a47-9c73af7346fc\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:09.126293 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:09.126271    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:09.402985 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:09.402960    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"]
Sep 10 16:05:09.404858 ip-10-0-139-11 kubenswrapper[2577]: W0910 16:05:09.404828    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf0fed77_0cfb_484c_8a47_9c73af7346fc.slice/crio-239248d5612412eb0e533b1de05a92972f5443909d39356fa19a0cd68f98a69b WatchSource:0}: Error finding container 239248d5612412eb0e533b1de05a92972f5443909d39356fa19a0cd68f98a69b: Status 404 returned error can't find the container with id 239248d5612412eb0e533b1de05a92972f5443909d39356fa19a0cd68f98a69b
Sep 10 16:05:09.448736 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:09.448710    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf" event={"ID":"bf0fed77-0cfb-484c-8a47-9c73af7346fc","Type":"ContainerStarted","Data":"239248d5612412eb0e533b1de05a92972f5443909d39356fa19a0cd68f98a69b"}
Sep 10 16:05:10.456548 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:10.456519    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf" event={"ID":"bf0fed77-0cfb-484c-8a47-9c73af7346fc","Type":"ContainerStarted","Data":"7045b9790594d8b03bb1921305c529944463574775cd75d2be3223fc98dd8011"}
Sep 10 16:05:10.456956 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:10.456618    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:05:10.480877 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:10.480829    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf" podStartSLOduration=2.480815324 podStartE2EDuration="2.480815324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:10.47905299 +0000 UTC m=+1954.093179896" watchObservedRunningTime="2026-09-10 16:05:10.480815324 +0000 UTC m=+1954.094942228"
Sep 10 16:05:21.464716 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:05:21.464682    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-6fprf"
Sep 10 16:07:01.286974 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.286897    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"]
Sep 10 16:07:01.290449 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.290430    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.303126 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.303096    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"]
Sep 10 16:07:01.408652 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.408617    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c494edb-6bb4-4141-b1a4-03376ebdf609-openshift-service-ca\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.408839 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.408728    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5wt9\" (UniqueName: \"kubernetes.io/projected/6c494edb-6bb4-4141-b1a4-03376ebdf609-kube-api-access-b5wt9\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.408839 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.408777    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c494edb-6bb4-4141-b1a4-03376ebdf609-maas-api-tls\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.408839 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.408799    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c494edb-6bb4-4141-b1a4-03376ebdf609-metrics-certs\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.509439 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.509399    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5wt9\" (UniqueName: \"kubernetes.io/projected/6c494edb-6bb4-4141-b1a4-03376ebdf609-kube-api-access-b5wt9\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.509439 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.509445    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c494edb-6bb4-4141-b1a4-03376ebdf609-maas-api-tls\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.509718 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.509467    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c494edb-6bb4-4141-b1a4-03376ebdf609-metrics-certs\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.509718 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.509487    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c494edb-6bb4-4141-b1a4-03376ebdf609-openshift-service-ca\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.510275 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.510203    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6c494edb-6bb4-4141-b1a4-03376ebdf609-openshift-service-ca\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.512163 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.512135    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6c494edb-6bb4-4141-b1a4-03376ebdf609-maas-api-tls\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.512267 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.512215    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6c494edb-6bb4-4141-b1a4-03376ebdf609-metrics-certs\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.517201 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.517176    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5wt9\" (UniqueName: \"kubernetes.io/projected/6c494edb-6bb4-4141-b1a4-03376ebdf609-kube-api-access-b5wt9\") pod \"maas-api-bcbc8bfdb-sbpwf\" (UID: \"6c494edb-6bb4-4141-b1a4-03376ebdf609\") " pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.604367 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.604332    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:01.879125 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:01.879027    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"]
Sep 10 16:07:01.886682 ip-10-0-139-11 kubenswrapper[2577]: W0910 16:07:01.886653    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c494edb_6bb4_4141_b1a4_03376ebdf609.slice/crio-93258511ec665f60102e9d8b0955a209b33c5c11240e27cd5e4186fa8827ac0d WatchSource:0}: Error finding container 93258511ec665f60102e9d8b0955a209b33c5c11240e27cd5e4186fa8827ac0d: Status 404 returned error can't find the container with id 93258511ec665f60102e9d8b0955a209b33c5c11240e27cd5e4186fa8827ac0d
Sep 10 16:07:02.057424 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:02.057385    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf" event={"ID":"6c494edb-6bb4-4141-b1a4-03376ebdf609","Type":"ContainerStarted","Data":"93258511ec665f60102e9d8b0955a209b33c5c11240e27cd5e4186fa8827ac0d"}
Sep 10 16:07:03.063371 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:03.063278    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf" event={"ID":"6c494edb-6bb4-4141-b1a4-03376ebdf609","Type":"ContainerStarted","Data":"6c8bcdf94e069d408245f487486b75136334adfcc6a801a3741ca3751098e788"}
Sep 10 16:07:03.063371 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:03.063357    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:03.082644 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:03.082584    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf" podStartSLOduration=1.375220927 podStartE2EDuration="2.082564325s" totalImagesPullingTime="707.343398ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:03.082309758 +0000 UTC m=+2066.696436655" watchObservedRunningTime="2026-09-10 16:07:03.082564325 +0000 UTC m=+2066.696691231"
Sep 10 16:07:09.075513 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:09.075482    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-bcbc8bfdb-sbpwf"
Sep 10 16:07:09.121702 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:09.121673    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-699674684b-pt4jc"]
Sep 10 16:07:09.121911 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:09.121888    2577 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc" podUID="b251bb68-9eda-487e-bb07-8d0ee793ff4f" containerName="maas-api" containerID="cri-o://8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0" gracePeriod=30
Sep 10 16:07:09.486394 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:09.486299    2577 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc" podUID="b251bb68-9eda-487e-bb07-8d0ee793ff4f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.47:8443/health\": dial tcp 10.134.0.47:8443: connect: connection refused"
Sep 10 16:07:10.369765 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.369721    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 16:07:10.493371 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.493278    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251bb68-9eda-487e-bb07-8d0ee793ff4f-openshift-service-ca\") pod \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") "
Sep 10 16:07:10.493371 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.493325    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgcnv\" (UniqueName: \"kubernetes.io/projected/b251bb68-9eda-487e-bb07-8d0ee793ff4f-kube-api-access-zgcnv\") pod \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") "
Sep 10 16:07:10.493371 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.493360    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-maas-api-tls\") pod \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") "
Sep 10 16:07:10.493683 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.493381    2577 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-metrics-certs\") pod \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\" (UID: \"b251bb68-9eda-487e-bb07-8d0ee793ff4f\") "
Sep 10 16:07:10.493787 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.493737    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b251bb68-9eda-487e-bb07-8d0ee793ff4f-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b251bb68-9eda-487e-bb07-8d0ee793ff4f" (UID: "b251bb68-9eda-487e-bb07-8d0ee793ff4f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:10.495857 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.495833    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b251bb68-9eda-487e-bb07-8d0ee793ff4f" (UID: "b251bb68-9eda-487e-bb07-8d0ee793ff4f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:07:10.496169 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.496139    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b251bb68-9eda-487e-bb07-8d0ee793ff4f" (UID: "b251bb68-9eda-487e-bb07-8d0ee793ff4f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:07:10.496169 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.496146    2577 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b251bb68-9eda-487e-bb07-8d0ee793ff4f-kube-api-access-zgcnv" (OuterVolumeSpecName: "kube-api-access-zgcnv") pod "b251bb68-9eda-487e-bb07-8d0ee793ff4f" (UID: "b251bb68-9eda-487e-bb07-8d0ee793ff4f"). InnerVolumeSpecName "kube-api-access-zgcnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:10.594501 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.594462    2577 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-maas-api-tls\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:07:10.594501 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.594491    2577 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b251bb68-9eda-487e-bb07-8d0ee793ff4f-metrics-certs\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:07:10.594501 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.594501    2577 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b251bb68-9eda-487e-bb07-8d0ee793ff4f-openshift-service-ca\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:07:10.594501 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:10.594510    2577 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgcnv\" (UniqueName: \"kubernetes.io/projected/b251bb68-9eda-487e-bb07-8d0ee793ff4f-kube-api-access-zgcnv\") on node \"ip-10-0-139-11.ec2.internal\" DevicePath \"\""
Sep 10 16:07:11.106332 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.106297    2577 generic.go:356] "Generic (PLEG): container finished" podID="b251bb68-9eda-487e-bb07-8d0ee793ff4f" containerID="8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0" exitCode=0
Sep 10 16:07:11.106332 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.106338    2577 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc"
Sep 10 16:07:11.106567 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.106351    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc" event={"ID":"b251bb68-9eda-487e-bb07-8d0ee793ff4f","Type":"ContainerDied","Data":"8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0"}
Sep 10 16:07:11.106567 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.106404    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-699674684b-pt4jc" event={"ID":"b251bb68-9eda-487e-bb07-8d0ee793ff4f","Type":"ContainerDied","Data":"376c7f1b6becdbca30663f24ff1768f854d8698165b49c8642df8c3a92e0a6b8"}
Sep 10 16:07:11.106567 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.106433    2577 scope.go:117] "RemoveContainer" containerID="8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0"
Sep 10 16:07:11.115490 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.115470    2577 scope.go:117] "RemoveContainer" containerID="8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0"
Sep 10 16:07:11.115777 ip-10-0-139-11 kubenswrapper[2577]: E0910 16:07:11.115739    2577 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0\": container with ID starting with 8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0 not found: ID does not exist" containerID="8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0"
Sep 10 16:07:11.115831 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.115791    2577 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0"} err="failed to get container status \"8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0\": rpc error: code = NotFound desc = could not find container \"8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0\": container with ID starting with 8c359025ec9656cc15a820b0df177fd89562030e827abe3420ebb169b5c8b6d0 not found: ID does not exist"
Sep 10 16:07:11.123472 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.123451    2577 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-699674684b-pt4jc"]
Sep 10 16:07:11.127387 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:11.127367    2577 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-699674684b-pt4jc"]
Sep 10 16:07:12.912470 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:12.912023    2577 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b251bb68-9eda-487e-bb07-8d0ee793ff4f" path="/var/lib/kubelet/pods/b251bb68-9eda-487e-bb07-8d0ee793ff4f/volumes"
Sep 10 16:07:31.581416 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:31.581377    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7676795785-b4j99_99bd3d28-a934-479d-a7d4-0a9abdd179fc/authorino/0.log"
Sep 10 16:07:36.978967 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:36.978932    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-59xd9_bc8f5d61-999a-46ab-9613-da4dd8616031/manager/0.log"
Sep 10 16:07:37.103025 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:37.102966    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gftql_fea22d52-da84-4c0a-82a9-6f2e7e9e555c/manager/0.log"
Sep 10 16:07:37.235094 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:37.235003    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569c478d95-qn5wh_8cf55be4-0755-416b-b4e3-a0633cd5d777/manager/0.log"
Sep 10 16:07:37.843363 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:37.843300    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-nr9h7_4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a/copy-manifests/0.log"
Sep 10 16:07:37.849740 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:37.849707    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-nr9h7_4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a/manager/0.log"
Sep 10 16:07:38.330517 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:38.330489    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-bcbc8bfdb-sbpwf_6c494edb-6bb4-4141-b1a4-03376ebdf609/maas-api/0.log"
Sep 10 16:07:38.674524 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:38.674436    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-kplx7_f56c561c-4da8-4e21-8fd0-3fef3671aa8b/postgres/0.log"
Sep 10 16:07:39.508002 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:39.507970    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7676795785-b4j99_99bd3d28-a934-479d-a7d4-0a9abdd179fc/authorino/0.log"
Sep 10 16:07:39.628345 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:39.628317    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5jwrc_18e6aad0-75ad-4a8a-9401-ff5378e30609/manager/0.log"
Sep 10 16:07:39.742421 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:39.742392    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-j42hr_c6def56d-78d9-4db4-9209-819e5349fbed/manager/0.log"
Sep 10 16:07:39.990029 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:39.989995    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6fprf_bf0fed77-0cfb-484c-8a47-9c73af7346fc/manager/0.log"
Sep 10 16:07:40.231174 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:40.231141    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-sppbt_53501126-73af-4b23-888c-e9d1660de15b/manager/0.log"
Sep 10 16:07:40.830542 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:40.830510    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5b58d48c-cktvx_f5fade1d-b1ca-48ab-a12f-90bdf2c895b0/kube-auth-proxy/0.log"
Sep 10 16:07:41.197830 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:41.197726    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zdst_92a2bd37-c4e6-4592-b933-db111b0fb2e3/payload-pre-processing/0.log"
Sep 10 16:07:41.805207 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:41.805178    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-zs5hj_8dac9953-e6c6-4281-92aa-26684f111031/main/0.log"
Sep 10 16:07:42.035357 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:42.035324    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-bcc9ff8d8-mvkfd_1b00eea2-d0cf-4bc9-9bca-a354c796952f/main/0.log"
Sep 10 16:07:48.236697 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.236662    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ccz5g/must-gather-884dh"]
Sep 10 16:07:48.237272 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.237205    2577 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b251bb68-9eda-487e-bb07-8d0ee793ff4f" containerName="maas-api"
Sep 10 16:07:48.237272 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.237224    2577 state_mem.go:107] "Deleted CPUSet assignment" podUID="b251bb68-9eda-487e-bb07-8d0ee793ff4f" containerName="maas-api"
Sep 10 16:07:48.237404 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.237316    2577 memory_manager.go:368] "RemoveStaleState removing state" podUID="b251bb68-9eda-487e-bb07-8d0ee793ff4f" containerName="maas-api"
Sep 10 16:07:48.240647 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.240625    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.242893 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.242872    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ccz5g\"/\"kube-root-ca.crt\""
Sep 10 16:07:48.243633 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.243615    2577 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-ccz5g\"/\"default-dockercfg-5k4tw\""
Sep 10 16:07:48.243633 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.243619    2577 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-ccz5g\"/\"openshift-service-ca.crt\""
Sep 10 16:07:48.247580 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.247559    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccz5g/must-gather-884dh"]
Sep 10 16:07:48.298881 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.298847    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f8433e94-a3ed-4949-9494-2f01217be50d-must-gather-output\") pod \"must-gather-884dh\" (UID: \"f8433e94-a3ed-4949-9494-2f01217be50d\") " pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.299026 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.298901    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nnzc\" (UniqueName: \"kubernetes.io/projected/f8433e94-a3ed-4949-9494-2f01217be50d-kube-api-access-5nnzc\") pod \"must-gather-884dh\" (UID: \"f8433e94-a3ed-4949-9494-2f01217be50d\") " pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.399662 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.399631    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f8433e94-a3ed-4949-9494-2f01217be50d-must-gather-output\") pod \"must-gather-884dh\" (UID: \"f8433e94-a3ed-4949-9494-2f01217be50d\") " pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.399855 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.399667    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nnzc\" (UniqueName: \"kubernetes.io/projected/f8433e94-a3ed-4949-9494-2f01217be50d-kube-api-access-5nnzc\") pod \"must-gather-884dh\" (UID: \"f8433e94-a3ed-4949-9494-2f01217be50d\") " pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.400030 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.400008    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/f8433e94-a3ed-4949-9494-2f01217be50d-must-gather-output\") pod \"must-gather-884dh\" (UID: \"f8433e94-a3ed-4949-9494-2f01217be50d\") " pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.408573 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.408551    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nnzc\" (UniqueName: \"kubernetes.io/projected/f8433e94-a3ed-4949-9494-2f01217be50d-kube-api-access-5nnzc\") pod \"must-gather-884dh\" (UID: \"f8433e94-a3ed-4949-9494-2f01217be50d\") " pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.550992 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.550916    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccz5g/must-gather-884dh"
Sep 10 16:07:48.826052 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.826023    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccz5g/must-gather-884dh"]
Sep 10 16:07:48.827026 ip-10-0-139-11 kubenswrapper[2577]: W0910 16:07:48.826985    2577 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf8433e94_a3ed_4949_9494_2f01217be50d.slice/crio-7bc2bd0b08033010dbdf49dbee79ca5a8848ed0cbc2acbf9cc658cf3f1e6bdde WatchSource:0}: Error finding container 7bc2bd0b08033010dbdf49dbee79ca5a8848ed0cbc2acbf9cc658cf3f1e6bdde: Status 404 returned error can't find the container with id 7bc2bd0b08033010dbdf49dbee79ca5a8848ed0cbc2acbf9cc658cf3f1e6bdde
Sep 10 16:07:48.828862 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:48.828844    2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:07:49.305389 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:49.305300    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccz5g/must-gather-884dh" event={"ID":"f8433e94-a3ed-4949-9494-2f01217be50d","Type":"ContainerStarted","Data":"7bc2bd0b08033010dbdf49dbee79ca5a8848ed0cbc2acbf9cc658cf3f1e6bdde"}
Sep 10 16:07:50.317128 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:50.317088    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccz5g/must-gather-884dh" event={"ID":"f8433e94-a3ed-4949-9494-2f01217be50d","Type":"ContainerStarted","Data":"9caa4350ea1d0ad86250182304e0d5e29388f4716fd10944cad0cfe9b68cec55"}
Sep 10 16:07:50.317583 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:50.317143    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccz5g/must-gather-884dh" event={"ID":"f8433e94-a3ed-4949-9494-2f01217be50d","Type":"ContainerStarted","Data":"b015c304e216b1a597b37ad3f3c96e5b29f094be148a647f89efe2d056a08f58"}
Sep 10 16:07:50.332167 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:50.332119    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ccz5g/must-gather-884dh" podStartSLOduration=1.382713715 podStartE2EDuration="2.332102238s" totalImagesPullingTime="949.388523ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:50.330776264 +0000 UTC m=+2113.944903170" watchObservedRunningTime="2026-09-10 16:07:50.332102238 +0000 UTC m=+2113.946229142"
Sep 10 16:07:51.393597 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:51.393563    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-j5nr8_5d4644fb-55db-4320-9d71-ca7b885427f5/global-pull-secret-syncer/0.log"
Sep 10 16:07:51.480307 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:51.480265    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-5d7px_04cdccb3-c41a-423d-8a73-ffd42a5ccaa9/konnectivity-agent/0.log"
Sep 10 16:07:51.576620 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:51.576586    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-11.ec2.internal_b183149671d7e63b663b38bcacd05dd5/haproxy/0.log"
Sep 10 16:07:56.284776 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:56.282859    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-7676795785-b4j99_99bd3d28-a934-479d-a7d4-0a9abdd179fc/authorino/0.log"
Sep 10 16:07:56.319402 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:56.319369    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-76b478bd78-5jwrc_18e6aad0-75ad-4a8a-9401-ff5378e30609/manager/0.log"
Sep 10 16:07:56.358557 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:56.358525    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-j42hr_c6def56d-78d9-4db4-9209-819e5349fbed/manager/0.log"
Sep 10 16:07:56.422188 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:56.422151    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-6fprf_bf0fed77-0cfb-484c-8a47-9c73af7346fc/manager/0.log"
Sep 10 16:07:56.516625 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:56.516580    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-sppbt_53501126-73af-4b23-888c-e9d1660de15b/manager/0.log"
Sep 10 16:07:58.268871 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:58.268844    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-85bbbbbf4-7vxk5_823f89ba-9c71-488a-b391-68a70217eea4/monitoring-plugin/0.log"
Sep 10 16:07:58.479604 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:58.479576    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4h8q_d7541db6-c688-4047-b38f-3d16ddfc476c/node-exporter/0.log"
Sep 10 16:07:58.507273 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:58.507207    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4h8q_d7541db6-c688-4047-b38f-3d16ddfc476c/kube-rbac-proxy/0.log"
Sep 10 16:07:58.539870 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:07:58.539784    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-w4h8q_d7541db6-c688-4047-b38f-3d16ddfc476c/init-textfile/0.log"
Sep 10 16:08:01.348039 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.347999    2577 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"]
Sep 10 16:08:01.353787 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.353736    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.360768 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.360715    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"]
Sep 10 16:08:01.435337 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.435300    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-sys\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.435617 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.435593    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-proc\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.435792 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.435776    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-podres\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.435992 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.435968    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sckss\" (UniqueName: \"kubernetes.io/projected/a3872393-b1ef-44d6-996e-9418eb386f59-kube-api-access-sckss\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.436112 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.436013    2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-lib-modules\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537549 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537514    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-sys\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537549 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537554    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-proc\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537798 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537593    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-podres\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537798 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537639    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-sys\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537798 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537684    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sckss\" (UniqueName: \"kubernetes.io/projected/a3872393-b1ef-44d6-996e-9418eb386f59-kube-api-access-sckss\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537798 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537716    2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-lib-modules\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537798 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537735    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-proc\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.537798 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537783    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-podres\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.538028 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.537865    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a3872393-b1ef-44d6-996e-9418eb386f59-lib-modules\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.546569 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.546528    2577 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sckss\" (UniqueName: \"kubernetes.io/projected/a3872393-b1ef-44d6-996e-9418eb386f59-kube-api-access-sckss\") pod \"perf-node-gather-daemonset-t5clf\" (UID: \"a3872393-b1ef-44d6-996e-9418eb386f59\") " pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.666984 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.666882    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75fb6787d5-2zngj_f0db9eda-9772-4212-8af5-d1d89a9b7c96/console/0.log"
Sep 10 16:08:01.668492 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.668463    2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:01.956332 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:01.956303    2577 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"]
Sep 10 16:08:02.415501 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:02.415458    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf" event={"ID":"a3872393-b1ef-44d6-996e-9418eb386f59","Type":"ContainerStarted","Data":"dd45ef0ef3adb08f151ed668b049cf66af54d655c1ed28bfedcc66b1edac639e"}
Sep 10 16:08:02.416000 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:02.415528    2577 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf" event={"ID":"a3872393-b1ef-44d6-996e-9418eb386f59","Type":"ContainerStarted","Data":"d48dbefd30f895cec21c8748ef1adf86db21c7a350e54b5a24ca2eba9ec778e3"}
Sep 10 16:08:02.416382 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:02.416356    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:02.434953 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:02.434898    2577 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf" podStartSLOduration=1.434879856 podStartE2EDuration="1.434879856s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:08:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:08:02.430215703 +0000 UTC m=+2126.044342609" watchObservedRunningTime="2026-09-10 16:08:02.434879856 +0000 UTC m=+2126.049006762"
Sep 10 16:08:03.593389 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:03.593361    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dp7tl_f195d304-9189-4b43-839c-25369d19155f/dns/0.log"
Sep 10 16:08:03.614314 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:03.614279    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dp7tl_f195d304-9189-4b43-839c-25369d19155f/kube-rbac-proxy/0.log"
Sep 10 16:08:03.725443 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:03.725407    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hm9kh_c0726f3e-2027-4c55-8f1a-7d9c73e46994/dns-node-resolver/0.log"
Sep 10 16:08:04.313080 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:04.313043    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jbtrv_9bf31f82-2e50-4346-ba50-f4b6edcd9a2b/node-ca/0.log"
Sep 10 16:08:05.620424 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:05.620389    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5b58d48c-cktvx_f5fade1d-b1ca-48ab-a12f-90bdf2c895b0/kube-auth-proxy/0.log"
Sep 10 16:08:05.730491 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:05.730457    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zdst_92a2bd37-c4e6-4592-b933-db111b0fb2e3/payload-pre-processing/0.log"
Sep 10 16:08:06.698166 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:06.698141    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-8zpnr_e5ff4d54-bda4-480e-9f05-1b4b63e8fc08/serve-healthcheck-canary/0.log"
Sep 10 16:08:07.300543 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:07.300513    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h27xd_38279413-0a92-4188-95cb-675c20eafd20/kube-rbac-proxy/0.log"
Sep 10 16:08:07.322276 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:07.322247    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h27xd_38279413-0a92-4188-95cb-675c20eafd20/exporter/0.log"
Sep 10 16:08:07.345100 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:07.345058    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-h27xd_38279413-0a92-4188-95cb-675c20eafd20/extractor/0.log"
Sep 10 16:08:09.444257 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:09.443532    2577 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-ccz5g/perf-node-gather-daemonset-t5clf"
Sep 10 16:08:09.477437 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:09.477395    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-59xd9_bc8f5d61-999a-46ab-9613-da4dd8616031/manager/0.log"
Sep 10 16:08:09.831660 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:09.831626    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gftql_fea22d52-da84-4c0a-82a9-6f2e7e9e555c/manager/0.log"
Sep 10 16:08:09.901570 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:09.901528    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-569c478d95-qn5wh_8cf55be4-0755-416b-b4e3-a0633cd5d777/manager/0.log"
Sep 10 16:08:10.179583 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:10.179504    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-nr9h7_4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a/manager/0.log"
Sep 10 16:08:10.199860 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:10.199827    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-9f4cd445f-nr9h7_4b00bcf8-b8dc-4ce4-94c8-c17c197bad8a/copy-manifests/0.log"
Sep 10 16:08:11.524194 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:11.524160    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-n6qpv_a243ec47-9a4e-4cfd-9ef5-6e74b818ba52/openshift-lws-operator/0.log"
Sep 10 16:08:12.342505 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:12.342481    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5b58d48c-cktvx_f5fade1d-b1ca-48ab-a12f-90bdf2c895b0/kube-auth-proxy/0.log"
Sep 10 16:08:12.430880 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:12.430848    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zdst_92a2bd37-c4e6-4592-b933-db111b0fb2e3/payload-pre-processing/0.log"
Sep 10 16:08:14.267746 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:14.267653    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b5b58d48c-cktvx_f5fade1d-b1ca-48ab-a12f-90bdf2c895b0/kube-auth-proxy/0.log"
Sep 10 16:08:14.349613 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:14.349572    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-2zdst_92a2bd37-c4e6-4592-b933-db111b0fb2e3/payload-pre-processing/0.log"
Sep 10 16:08:18.414195 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.414161    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzwtb_d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac/kube-multus-additional-cni-plugins/0.log"
Sep 10 16:08:18.433716 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.433686    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzwtb_d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac/egress-router-binary-copy/0.log"
Sep 10 16:08:18.453205 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.453182    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzwtb_d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac/cni-plugins/0.log"
Sep 10 16:08:18.472814 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.472792    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzwtb_d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac/bond-cni-plugin/0.log"
Sep 10 16:08:18.491602 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.491581    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzwtb_d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac/routeoverride-cni/0.log"
Sep 10 16:08:18.510746 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.510721    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzwtb_d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac/whereabouts-cni-bincopy/0.log"
Sep 10 16:08:18.539242 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.539221    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-vzwtb_d1e67c7f-d563-43d1-96b7-1cdeb9f7c1ac/whereabouts-cni/0.log"
Sep 10 16:08:18.786904 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.786806    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-n52xc_e1ac5f46-ec24-4285-8753-95bbb3bc7d17/kube-multus/0.log"
Sep 10 16:08:18.949682 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.949653    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qpkv8_08fb1852-cdb8-4abb-97f6-f85fdb8f4651/network-metrics-daemon/0.log"
Sep 10 16:08:18.969400 ip-10-0-139-11 kubenswrapper[2577]: I0910 16:08:18.969363    2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qpkv8_08fb1852-cdb8-4abb-97f6-f85fdb8f4651/kube-rbac-proxy/0.log"