Sep 07 22:32:13.412640 ip-10-0-137-80 systemd[1]: Starting Kubernetes Kubelet...
Sep 07 22:32:13.804153 ip-10-0-137-80 kubenswrapper[2582]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:32:13.804153 ip-10-0-137-80 kubenswrapper[2582]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 07 22:32:13.804153 ip-10-0-137-80 kubenswrapper[2582]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:32:13.804153 ip-10-0-137-80 kubenswrapper[2582]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 07 22:32:13.804153 ip-10-0-137-80 kubenswrapper[2582]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 07 22:32:13.805955 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.805863    2582 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 07 22:32:13.810926 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810899    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:13.810926 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810921    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:13.810926 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810925    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:13.810926 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810929    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:13.810926 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810932    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:13.810926 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810935    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810938    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810941    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810946    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810951    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810955    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810959    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810962    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810965    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810968    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810971    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810974    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810977    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810980    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810983    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810985    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810989    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810992    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810995    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:13.811157 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.810997    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811007    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811010    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811013    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811015    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811018    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811021    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811023    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811026    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811028    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811030    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811033    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811035    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811038    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811040    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811043    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811046    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811048    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811051    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811054    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:13.811677 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811056    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811059    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811061    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811064    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811066    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811069    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811071    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811074    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811076    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811079    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811082    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811084    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811087    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811090    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811093    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811095    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811098    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811100    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811103    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:13.812168 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811107    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811111    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811114    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811116    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811119    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811121    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811124    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811127    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811130    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811133    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811136    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811138    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811141    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811143    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811146    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811148    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811152    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811155    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811157    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811159    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:13.812640 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811164    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811168    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811171    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811173    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811176    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811178    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811181    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811183    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811186    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811188    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811191    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811193    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811196    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811198    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811201    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811203    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811904    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811911    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811914    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811917    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:13.813127 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811921    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811925    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811927    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811930    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811933    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811937    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811941    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811944    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811947    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811950    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811952    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811955    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811958    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811962    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811965    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811967    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811970    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811973    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811975    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:13.813626 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811978    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811980    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811983    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811985    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811988    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811991    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811993    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811996    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.811998    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812001    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812003    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812006    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812008    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812012    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812014    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812017    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812020    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812022    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812025    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:13.814144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812029    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812033    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812036    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812038    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812041    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812043    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812046    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812049    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812052    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812055    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812057    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812060    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812062    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812065    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812069    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812073    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812076    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812079    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812081    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:13.814631 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812084    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812087    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812089    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812091    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812094    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812097    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812099    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812104    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812107    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812109    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812112    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812114    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812117    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812120    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812123    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812125    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812128    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812131    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812133    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812136    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:13.815182 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812138    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812141    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812144    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812147    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812150    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812153    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812155    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812158    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812161    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812163    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812166    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812168    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812171    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812174    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812177    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812180    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812183    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812186    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812621    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812628    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:13.815702 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812632    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812635    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812638    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812641    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812644    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812648    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812662    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812666    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812669    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812672    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812675    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812678    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812680    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812684    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812689    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812692    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812695    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812698    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812700    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:13.816202 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812703    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812706    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812708    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812711    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812714    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812717    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812719    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812722    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812725    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812727    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812730    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812733    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812735    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812738    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812740    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812743    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812745    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812748    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812751    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:13.816688 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812753    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812756    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812759    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812763    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812766    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812771    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812775    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812779    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812785    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812791    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812795    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812799    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812803    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812806    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812809    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812811    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812815    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812817    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812820    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812823    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:13.817149 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812825    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812828    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812831    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812833    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812836    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812839    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812842    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812844    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812847    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812850    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812852    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812854    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812857    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812859    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812864    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812868    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812871    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812873    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812876    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:13.817660 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812879    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812881    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812884    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812887    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812889    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812894    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812897    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812899    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812902    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812904    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812907    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812910    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812913    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812915    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812918    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812920    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812923    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812926    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812928    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.812930    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813006    2582 flags.go:64] FLAG: --address="0.0.0.0"
Sep 07 22:32:13.818119 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813045    2582 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813057    2582 flags.go:64] FLAG: --anonymous-auth="true"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813062    2582 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813066    2582 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813069    2582 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813074    2582 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813078    2582 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813082    2582 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813085    2582 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813088    2582 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813092    2582 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813095    2582 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813098    2582 flags.go:64] FLAG: --cgroup-root=""
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813101    2582 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813106    2582 flags.go:64] FLAG: --client-ca-file=""
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813109    2582 flags.go:64] FLAG: --cloud-provider="external"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813114    2582 flags.go:64] FLAG: --cluster-dns="[]"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813118    2582 flags.go:64] FLAG: --cluster-domain=""
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813121    2582 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813124    2582 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813127    2582 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813130    2582 flags.go:64] FLAG: --container-log-max-files="5"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813134    2582 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 07 22:32:13.818648 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813137    2582 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813140    2582 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813144    2582 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813147    2582 flags.go:64] FLAG: --contention-profiling="false"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813150    2582 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813153    2582 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813156    2582 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813159    2582 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813163    2582 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813166    2582 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813169    2582 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813172    2582 flags.go:64] FLAG: --enable-load-reader="false"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813175    2582 flags.go:64] FLAG: --enable-server="true"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813177    2582 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813182    2582 flags.go:64] FLAG: --event-burst="100"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813185    2582 flags.go:64] FLAG: --event-qps="50"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813188    2582 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813191    2582 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813194    2582 flags.go:64] FLAG: --eviction-hard=""
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813198    2582 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813200    2582 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813204    2582 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813206    2582 flags.go:64] FLAG: --eviction-soft=""
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813211    2582 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813214    2582 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 07 22:32:13.819189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813221    2582 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813224    2582 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813227    2582 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813230    2582 flags.go:64] FLAG: --fail-swap-on="true"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813233    2582 flags.go:64] FLAG: --feature-gates=""
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813237    2582 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813240    2582 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813243    2582 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813246    2582 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813249    2582 flags.go:64] FLAG: --healthz-port="10248"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813252    2582 flags.go:64] FLAG: --help="false"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813255    2582 flags.go:64] FLAG: --hostname-override="ip-10-0-137-80.ec2.internal"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813258    2582 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813261    2582 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813264    2582 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813267    2582 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813270    2582 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813273    2582 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813276    2582 flags.go:64] FLAG: --image-service-endpoint=""
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813279    2582 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813293    2582 flags.go:64] FLAG: --kube-api-burst="100"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813296    2582 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813299    2582 flags.go:64] FLAG: --kube-api-qps="50"
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813302    2582 flags.go:64] FLAG: --kube-reserved=""
Sep 07 22:32:13.819902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813305    2582 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813308    2582 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813311    2582 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813314    2582 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813317    2582 flags.go:64] FLAG: --lock-file=""
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813320    2582 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813324    2582 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813327    2582 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813334    2582 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813338    2582 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813341    2582 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813344    2582 flags.go:64] FLAG: --logging-format="text"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813346    2582 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813350    2582 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813353    2582 flags.go:64] FLAG: --manifest-url=""
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813355    2582 flags.go:64] FLAG: --manifest-url-header=""
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813360    2582 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813363    2582 flags.go:64] FLAG: --max-open-files="1000000"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813367    2582 flags.go:64] FLAG: --max-pods="110"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813371    2582 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813374    2582 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813377    2582 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813381    2582 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813384    2582 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813387    2582 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 07 22:32:13.820503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813390    2582 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813402    2582 flags.go:64] FLAG: --node-status-max-images="50"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813406    2582 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813408    2582 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813411    2582 flags.go:64] FLAG: --pod-cidr=""
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813414    2582 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813420    2582 flags.go:64] FLAG: --pod-manifest-path=""
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813423    2582 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813426    2582 flags.go:64] FLAG: --pods-per-core="0"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813429    2582 flags.go:64] FLAG: --port="10250"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813432    2582 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813435    2582 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0820872cda5c5bbf2"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813439    2582 flags.go:64] FLAG: --qos-reserved=""
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813443    2582 flags.go:64] FLAG: --read-only-port="10255"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813446    2582 flags.go:64] FLAG: --register-node="true"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813449    2582 flags.go:64] FLAG: --register-with-taints=""
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813454    2582 flags.go:64] FLAG: --registry-burst="10"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813458    2582 flags.go:64] FLAG: --registry-qps="5"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813461    2582 flags.go:64] FLAG: --reserved-cpus=""
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813464    2582 flags.go:64] FLAG: --reserved-memory=""
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813468    2582 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813471    2582 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813474    2582 flags.go:64] FLAG: --rotate-certificates="false"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813477    2582 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813480    2582 flags.go:64] FLAG: --runonce="false"
Sep 07 22:32:13.821131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813483    2582 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813486    2582 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813489    2582 flags.go:64] FLAG: --seccomp-default="false"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813491    2582 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813494    2582 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813498    2582 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813501    2582 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813504    2582 flags.go:64] FLAG: --storage-driver-password="root"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813507    2582 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813510    2582 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813513    2582 flags.go:64] FLAG: --storage-driver-user="root"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813516    2582 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813519    2582 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813522    2582 flags.go:64] FLAG: --system-cgroups=""
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813524    2582 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813530    2582 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813532    2582 flags.go:64] FLAG: --tls-cert-file=""
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813535    2582 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813539    2582 flags.go:64] FLAG: --tls-min-version=""
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813542    2582 flags.go:64] FLAG: --tls-private-key-file=""
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813546    2582 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813549    2582 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813552    2582 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813556    2582 flags.go:64] FLAG: --v="2"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813560    2582 flags.go:64] FLAG: --version="false"
Sep 07 22:32:13.821779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813565    2582 flags.go:64] FLAG: --vmodule=""
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813568    2582 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813571    2582 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813692    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813695    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813698    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813701    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813704    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813707    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813709    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813712    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813723    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813726    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813728    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813731    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813734    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813737    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813739    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813742    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813745    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:13.822475 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813747    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813750    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813753    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813755    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813758    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813760    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813763    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813767    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813770    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813772    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813776    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813779    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813781    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813786    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813790    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813793    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813796    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813799    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813801    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:13.822966 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813804    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813806    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813809    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813812    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813814    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813818    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813820    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813823    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813825    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813828    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813830    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813833    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813836    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813838    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813841    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813843    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813846    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813849    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813851    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:13.823483 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813854    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813858    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813860    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813863    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813867    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813870    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813872    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813875    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813878    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813881    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813884    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813886    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813889    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813891    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813894    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813897    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813899    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813902    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813904    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813907    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:13.823949 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813909    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813912    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813914    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813917    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813919    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813922    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813924    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813927    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813929    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813932    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813934    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813937    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813939    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813943    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813946    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813948    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813952    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813956    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813959    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813962    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:13.824439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813964    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813969    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813972    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.813976    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.813984    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.821159    2582 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.821176    2582 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.821840    2582 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.821853    2582 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821911    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821917    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821922    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821925    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821928    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:13.824937 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821932    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821935    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821937    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821940    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821943    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821946    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821949    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821951    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821954    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821956    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821959    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821962    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821964    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821968    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821972    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821975    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821978    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821981    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821984    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:13.825327 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821987    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821990    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821992    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821995    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.821998    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822004    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822007    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822010    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822012    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822015    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822017    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822020    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822023    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822025    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822028    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822031    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822033    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822036    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822038    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822041    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:13.825802 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822043    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822046    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822048    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822051    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822053    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822056    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822058    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822061    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822064    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822067    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822070    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822073    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822075    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822078    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822080    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822083    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822085    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822088    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822092    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822095    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822098    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:13.826388 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822101    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822104    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822106    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822109    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822112    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822114    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822117    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822119    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822122    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822124    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822127    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822129    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822132    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822135    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822137    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822140    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822142    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822145    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822147    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822150    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:13.826943 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822153    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822155    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822158    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822161    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822164    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822168    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822172    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822175    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822177    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822180    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822183    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822186    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822188    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822191    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.822196    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:32:13.827449 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822324    2582 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822331    2582 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822333    2582 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822337    2582 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822339    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822342    2582 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822344    2582 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822347    2582 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822350    2582 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822352    2582 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822355    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822357    2582 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822360    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822362    2582 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822365    2582 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822367    2582 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822370    2582 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822373    2582 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822376    2582 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822378    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 07 22:32:13.827820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822381    2582 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822383    2582 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822385    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822388    2582 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822391    2582 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822393    2582 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822396    2582 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822398    2582 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822401    2582 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822404    2582 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822406    2582 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822409    2582 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822411    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822413    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822416    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822418    2582 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822421    2582 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822423    2582 feature_gate.go:328] unrecognized feature gate: Example
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822426    2582 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822428    2582 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 07 22:32:13.828330 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822431    2582 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822433    2582 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822436    2582 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822438    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822441    2582 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822443    2582 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822446    2582 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822448    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822451    2582 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822453    2582 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822456    2582 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822460    2582 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822464    2582 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822467    2582 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822470    2582 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822473    2582 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822477    2582 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822481    2582 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 07 22:32:13.828819 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822484    2582 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822487    2582 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822490    2582 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822493    2582 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822496    2582 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822498    2582 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822501    2582 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822504    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822506    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822509    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822511    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822514    2582 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822516    2582 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822519    2582 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822522    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822524    2582 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822526    2582 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822529    2582 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822531    2582 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822534    2582 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 07 22:32:13.829266 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822536    2582 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822539    2582 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822541    2582 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822544    2582 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822546    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822549    2582 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822551    2582 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822554    2582 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822556    2582 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822559    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822561    2582 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822564    2582 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822567    2582 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822570    2582 feature_gate.go:328] unrecognized feature gate: Example2
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822572    2582 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822575    2582 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822579    2582 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822582    2582 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822585    2582 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822588    2582 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 07 22:32:13.829908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:13.822591    2582 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 07 22:32:13.830704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.822595    2582 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 07 22:32:13.830704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.822710    2582 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 07 22:32:13.830704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.824884    2582 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 07 22:32:13.830704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.825871    2582 server.go:1019] "Starting client certificate rotation"
Sep 07 22:32:13.830704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.825972    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 22:32:13.830704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.826002    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 07 22:32:13.830704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.828119    2582 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 22:32:13.831337 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.831314    2582 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 07 22:32:13.840053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.840036    2582 log.go:25] "Validated CRI v1 runtime API"
Sep 07 22:32:13.843078 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.843061    2582 log.go:25] "Validated CRI v1 image API"
Sep 07 22:32:13.844024 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.844002    2582 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 07 22:32:13.850007 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.849983    2582 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 82ffcb2e-0d89-497a-a13b-2615fdd3f679:/dev/nvme0n1p3 dbd00db7-3eea-4b76-9c13-02ea3394c4ba:/dev/nvme0n1p4]
Sep 07 22:32:13.850102 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.850003    2582 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 07 22:32:13.855304 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.855167    2582 manager.go:216] Machine: {Timestamp:2026-09-07 22:32:13.853831736 +0000 UTC m=+0.319988722 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101178 MemoryCapacity:33164484608 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec214f7c39a62e8c8d4bb0427c983212 SystemUUID:ec214f7c-39a6-2e8c-8d4b-b0427c983212 BootID:c93b590b-e2d9-41e4-b193-d8729bc91810 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582242304 Type:vfs Inodes:4048399 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582242304 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:38:c1:af:29:57 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:38:c1:af:29:57 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:46:69:b5:7e:1c:30 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164484608 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 07 22:32:13.855304 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.855300    2582 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 07 22:32:13.855424 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.855387    2582 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 07 22:32:13.858348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.858318    2582 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 07 22:32:13.858509 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.858350    2582 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-80.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 07 22:32:13.859120 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.859109    2582 topology_manager.go:138] "Creating topology manager with none policy"
Sep 07 22:32:13.859153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.859123    2582 container_manager_linux.go:309] "Creating device plugin manager"
Sep 07 22:32:13.859153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.859131    2582 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 22:32:13.859690 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.859680    2582 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 07 22:32:13.860455 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.860436    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 22:32:13.860855 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.860844    2582 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 07 22:32:13.863956 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.863943    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 22:32:13.864074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864064    2582 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 07 22:32:13.864117 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864094    2582 kubelet.go:486] "Attempting to sync node with API server"
Sep 07 22:32:13.864117 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864104    2582 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 07 22:32:13.864117 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864115    2582 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 07 22:32:13.864221 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864125    2582 kubelet.go:398] "Adding apiserver pod source"
Sep 07 22:32:13.864221 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864135    2582 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 07 22:32:13.864544 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864533    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 22:32:13.864579 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.864553    2582 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 07 22:32:13.867615 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.867598    2582 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 07 22:32:13.871008 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.870986    2582 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 07 22:32:13.871113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.871032    2582 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 07 22:32:13.873091 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873073    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 07 22:32:13.873175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873108    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 07 22:32:13.873175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873120    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 07 22:32:13.873175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873129    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 07 22:32:13.873175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873142    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 07 22:32:13.873175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873152    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 07 22:32:13.873175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873172    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 07 22:32:13.873405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873181    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 07 22:32:13.873405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873192    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 07 22:32:13.873405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873201    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 07 22:32:13.873405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873214    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 07 22:32:13.873405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.873223    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 07 22:32:13.874215 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.874203    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 07 22:32:13.874275 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.874218    2582 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 07 22:32:13.875877 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:13.875847    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-80.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 07 22:32:13.875990 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:13.875856    2582 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 07 22:32:13.876659 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.876630    2582 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-80.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 07 22:32:13.878277 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.878261    2582 server.go:1268] "Started kubelet"
Sep 07 22:32:13.878388 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.878358    2582 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 07 22:32:13.878449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.878402    2582 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 07 22:32:13.878491 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.878468    2582 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 07 22:32:13.879217 ip-10-0-137-80 systemd[1]: Started Kubernetes Kubelet.
Sep 07 22:32:13.879830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.879810    2582 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 07 22:32:13.880372 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.880344    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-89rjp"
Sep 07 22:32:13.880372 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.880370    2582 server.go:310] "Adding debug handlers to kubelet server"
Sep 07 22:32:13.885807 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:13.885770    2582 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 07 22:32:13.886090 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:13.885223    2582 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-80.ec2.internal.18d32a9db7cecba7  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-80.ec2.internal,UID:ip-10-0-137-80.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-80.ec2.internal,},FirstTimestamp:2026-09-07 22:32:13.878234023 +0000 UTC m=+0.344391008,LastTimestamp:2026-09-07 22:32:13.878234023 +0000 UTC m=+0.344391008,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-80.ec2.internal,}"
Sep 07 22:32:13.886774 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.886758    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-89rjp"
Sep 07 22:32:13.889685 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.889669    2582 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 07 22:32:13.890250 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.890236    2582 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 07 22:32:13.890414 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:13.890394    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:13.890991 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.890967    2582 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 07 22:32:13.891070 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891059    2582 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 07 22:32:13.891131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891073    2582 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 07 22:32:13.891194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891175    2582 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 07 22:32:13.891194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891191    2582 factory.go:55] Registering systemd factory
Sep 07 22:32:13.891194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891197    2582 factory.go:223] Registration of the systemd container factory successfully
Sep 07 22:32:13.891406 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891258    2582 reconstruct.go:97] "Volume reconstruction finished"
Sep 07 22:32:13.891406 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891268    2582 reconciler.go:29] "Reconciler: start to sync state"
Sep 07 22:32:13.891503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891424    2582 factory.go:153] Registering CRI-O factory
Sep 07 22:32:13.891503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891437    2582 factory.go:223] Registration of the crio container factory successfully
Sep 07 22:32:13.891503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891455    2582 factory.go:103] Registering Raw factory
Sep 07 22:32:13.891503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891472    2582 manager.go:1196] Started watching for new ooms in manager
Sep 07 22:32:13.891904 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.891888    2582 manager.go:349] Starting recovery of all containers
Sep 07 22:32:13.897001 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.896976    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:13.900415 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:13.900388    2582 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-137-80.ec2.internal\" not found" node="ip-10-0-137-80.ec2.internal"
Sep 07 22:32:13.905755 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.905584    2582 manager.go:354] Recovery completed
Sep 07 22:32:13.910041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.910027    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:13.911839 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.911819    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:13.911906 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.911856    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:13.911906 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.911871    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:13.912457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.912434    2582 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 07 22:32:13.912457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.912446    2582 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 07 22:32:13.912570 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.912461    2582 state_mem.go:36] "Initialized new in-memory state store"
Sep 07 22:32:13.912570 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.912494    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:13.913231 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.913217    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:13.913328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.913245    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:13.913328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:13.913255    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:13.990583 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:13.990542    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.051082 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.051063    2582 policy_none.go:49] "None policy: Start"
Sep 07 22:32:14.051130 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.051090    2582 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 07 22:32:14.051130 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.051103    2582 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 07 22:32:14.052201 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.052190    2582 policy_none.go:47] "Start"
Sep 07 22:32:14.081903 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.081866    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 07 22:32:14.083010 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.082994    2582 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 07 22:32:14.083094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.083061    2582 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 07 22:32:14.083094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.083090    2582 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 07 22:32:14.083206 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.083188    2582 kubelet.go:2472] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 07 22:32:14.085308 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.085274    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:14.087056 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.087042    2582 manager.go:337] "Starting Device Plugin manager"
Sep 07 22:32:14.087139 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.087084    2582 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 07 22:32:14.087139 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.087098    2582 server.go:87] "Starting device plugin registration server"
Sep 07 22:32:14.087478 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.087465    2582 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 07 22:32:14.087551 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.087480    2582 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 07 22:32:14.087608 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.087580    2582 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 07 22:32:14.087671 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.087658    2582 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 07 22:32:14.087671 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.087671    2582 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 07 22:32:14.088185 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.088166    2582 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 07 22:32:14.088259 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.088210    2582 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.184134 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.184066    2582 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal"]
Sep 07 22:32:14.184316 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.184177    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:14.185245 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.185227    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:14.185325 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.185259    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:14.185325 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.185275    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:14.186586 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.186572    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:14.186701 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.186685    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.186752 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.186719    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:14.187498 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.187479    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:14.187602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.187499    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:14.187602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.187524    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:14.187602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.187537    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:14.187602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.187506    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:14.187602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.187587    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:14.187812 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.187660    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:14.188439 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.188421    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:14.188525 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.188447    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:14.188525 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.188459    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:14.188525 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.188482    2582 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.188757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.188742    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.188831 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.188772    2582 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 07 22:32:14.189502 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.189486    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientMemory"
Sep 07 22:32:14.189584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.189511    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasNoDiskPressure"
Sep 07 22:32:14.189584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.189523    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeHasSufficientPID"
Sep 07 22:32:14.192573 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.192553    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5625e0c08c7af636ceb9734794ed4c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal\" (UID: \"5625e0c08c7af636ceb9734794ed4c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.192646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.192588    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5625e0c08c7af636ceb9734794ed4c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal\" (UID: \"5625e0c08c7af636ceb9734794ed4c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.192646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.192612    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbd0e3824c495bf54791857c054ed650-config\") pod \"kube-apiserver-proxy-ip-10-0-137-80.ec2.internal\" (UID: \"bbd0e3824c495bf54791857c054ed650\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.193333 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.193320    2582 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.193398 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.193339    2582 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-80.ec2.internal\": node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.203887 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.203864    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-80.ec2.internal\" not found" node="ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.207632 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.207614    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.208067 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.208053    2582 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-80.ec2.internal\" not found" node="ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.292860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.292829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5625e0c08c7af636ceb9734794ed4c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal\" (UID: \"5625e0c08c7af636ceb9734794ed4c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.292860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.292858    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5625e0c08c7af636ceb9734794ed4c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal\" (UID: \"5625e0c08c7af636ceb9734794ed4c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.293055 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.292875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbd0e3824c495bf54791857c054ed650-config\") pod \"kube-apiserver-proxy-ip-10-0-137-80.ec2.internal\" (UID: \"bbd0e3824c495bf54791857c054ed650\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.293055 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.292907    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/bbd0e3824c495bf54791857c054ed650-config\") pod \"kube-apiserver-proxy-ip-10-0-137-80.ec2.internal\" (UID: \"bbd0e3824c495bf54791857c054ed650\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.293055 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.292916    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/5625e0c08c7af636ceb9734794ed4c2f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal\" (UID: \"5625e0c08c7af636ceb9734794ed4c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.293055 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.292951    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5625e0c08c7af636ceb9734794ed4c2f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal\" (UID: \"5625e0c08c7af636ceb9734794ed4c2f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.307845 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.307819    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.408674 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.408578    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.506190 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.506160    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.508870 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.508849    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.511468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.511449    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:14.610142 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.610097    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.711259 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.711186    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.812372 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.812340    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.826203 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.826175    2582 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 07 22:32:14.826383 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.826359    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 22:32:14.826426 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.826370    2582 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 07 22:32:14.891496 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.891447    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-06 22:27:14 +0000 UTC" deadline="2028-02-05 04:40:33.77106347 +0000 UTC"
Sep 07 22:32:14.891496 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.891487    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12366h8m18.879580209s"
Sep 07 22:32:14.913471 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:14.913436    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:14.970663 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:14.970546    2582 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:15.014430 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:15.014374    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:15.080224 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:15.080077    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5625e0c08c7af636ceb9734794ed4c2f.slice/crio-bc7e09c7420d3ae53f5e419247c7d9f7cf9b177964bf05bd365478b64debb279 WatchSource:0}: Error finding container bc7e09c7420d3ae53f5e419247c7d9f7cf9b177964bf05bd365478b64debb279: Status 404 returned error can't find the container with id bc7e09c7420d3ae53f5e419247c7d9f7cf9b177964bf05bd365478b64debb279
Sep 07 22:32:15.080836 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:15.080808    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd0e3824c495bf54791857c054ed650.slice/crio-6eae1f73c57000db0446c1999a5c8618e9a74c1e331c306f1f7ced1be6bebcc1 WatchSource:0}: Error finding container 6eae1f73c57000db0446c1999a5c8618e9a74c1e331c306f1f7ced1be6bebcc1: Status 404 returned error can't find the container with id 6eae1f73c57000db0446c1999a5c8618e9a74c1e331c306f1f7ced1be6bebcc1
Sep 07 22:32:15.085786 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.085767    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:32:15.086632 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.086534    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal" event={"ID":"5625e0c08c7af636ceb9734794ed4c2f","Type":"ContainerStarted","Data":"bc7e09c7420d3ae53f5e419247c7d9f7cf9b177964bf05bd365478b64debb279"}
Sep 07 22:32:15.088001 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.087873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal" event={"ID":"bbd0e3824c495bf54791857c054ed650","Type":"ContainerStarted","Data":"6eae1f73c57000db0446c1999a5c8618e9a74c1e331c306f1f7ced1be6bebcc1"}
Sep 07 22:32:15.114523 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:15.114477    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:15.215611 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:15.215562    2582 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-80.ec2.internal\" not found"
Sep 07 22:32:15.299535 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.299456    2582 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:15.391383 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.391350    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:15.402315 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.402272    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 22:32:15.403611 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.403583    2582 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal"
Sep 07 22:32:15.412029 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.412007    2582 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 07 22:32:15.865485 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.865347    2582 apiserver.go:52] "Watching apiserver"
Sep 07 22:32:15.873413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.873303    2582 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 07 22:32:15.876443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.876397    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-kx45q","openshift-multus/multus-rjgxt","openshift-multus/network-metrics-daemon-4sqns","kube-system/global-pull-secret-syncer-22swc","kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal","openshift-dns/node-resolver-62ddd","openshift-network-diagnostics/network-check-target-vrxqj","openshift-network-operator/iptables-alerter-gbd66","openshift-ovn-kubernetes/ovnkube-node-h6c9s","kube-system/konnectivity-agent-cj55w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp","openshift-cluster-node-tuning-operator/tuned-pcvdb","openshift-image-registry/node-ca-fljbl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal"]
Sep 07 22:32:15.878276 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.878251    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:15.878417 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:15.878339    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:15.879588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.879560    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.880999 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.880980    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:15.881103 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:15.881062    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:15.882258 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.882027    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:15.882258 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:15.882093    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:15.882452 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.882313    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 07 22:32:15.882452 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.882330    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 07 22:32:15.882560 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.882516    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 07 22:32:15.882638 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.882622    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-grvkj\""
Sep 07 22:32:15.883810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.883619    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:15.885276 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.885065    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:15.885792 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.885770    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:32:15.886865 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.886841    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 07 22:32:15.887069 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.887050    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-w9cms\""
Sep 07 22:32:15.887240 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.887220    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 07 22:32:15.888435 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.887691    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.888435 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.887766    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 07 22:32:15.888435 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.888206    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r769h\""
Sep 07 22:32:15.888680 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.888659    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 07 22:32:15.890616 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.890596    2582 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 07 22:32:15.893098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.891565    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 07 22:32:15.893098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.891608    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 07 22:32:15.893098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.891640    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mb2sz\""
Sep 07 22:32:15.893098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.891871    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:15.893098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.891976    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 07 22:32:15.893098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.892345    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 07 22:32:15.893098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.892562    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 07 22:32:15.894763 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.894490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 07 22:32:15.894763 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.894495    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.895279 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.895258    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-fw4kc\""
Sep 07 22:32:15.895500 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.895481    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 07 22:32:15.895573 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.895548    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 07 22:32:15.895642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.895490    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 07 22:32:15.896099 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.896079    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:15.896995 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.896974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 07 22:32:15.897920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.897514    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-lclj8\""
Sep 07 22:32:15.897920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.897579    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 07 22:32:15.897920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.897777    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 07 22:32:15.899109 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.898465    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:15.899109 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.898854    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 07 22:32:15.900845 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.900826    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 07 22:32:15.900963 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.900942    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 07 22:32:15.901078 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.901061    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-q92qn\""
Sep 07 22:32:15.901657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.901633    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 07 22:32:15.901798 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.901773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-ovn\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.901872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.901808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a679bbf7-133d-425c-959d-8eb901ecdcf4-dbus\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:15.901872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.901836    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:15.901872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.901859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-cnibin\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.902031 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.901935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-cni-netd\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.902263 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvhg5\" (UniqueName: \"kubernetes.io/projected/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-kube-api-access-wvhg5\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:15.902402 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902288    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-cni-bin\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.902402 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902313    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-cni-multus\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.902402 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902356    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-netns\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.902402 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902382    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85c6s\" (UniqueName: \"kubernetes.io/projected/eef39637-cc2e-48ec-866a-101b11cd6329-kube-api-access-85c6s\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-systemd\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902457    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902480    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-var-lib-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902521    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-conf-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902546    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-kubelet\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902584    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-socket-dir-parent\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902614    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:15.902646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-slash\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902657    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovnkube-script-lib\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902671    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7zrdn\""
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902674    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/434fae71-2d34-4e71-b384-83aa51cf0a28-serviceca\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-log-socket\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902780    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovn-node-metrics-cert\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902808    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovnkube-config\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-device-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902879    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-systemd-units\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902929    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h27m2\" (UniqueName: \"kubernetes.io/projected/dbb33cfb-6569-441c-aaf6-045291b36801-kube-api-access-h27m2\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.902976    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-system-cni-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903005    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-kubelet\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903035    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-run-netns\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-hosts-file\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:15.903115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903102    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-multus-certs\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903126    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903159    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-tmp-dir\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903259    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-etc-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903291    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-env-overrides\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nn574\" (UniqueName: \"kubernetes.io/projected/434fae71-2d34-4e71-b384-83aa51cf0a28-kube-api-access-nn574\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903368    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47c85415-51eb-43b0-85d0-9ad397371ca2-iptables-alerter-script\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903381    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903398    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-socket-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbrhh\" (UniqueName: \"kubernetes.io/projected/47c85415-51eb-43b0-85d0-9ad397371ca2-kube-api-access-cbrhh\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903475    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-cni-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903500    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef39637-cc2e-48ec-866a-101b11cd6329-cni-binary-copy\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-hostroot\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903602    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef39637-cc2e-48ec-866a-101b11cd6329-multus-daemon-config\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903643    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-k8s-cni-cncf-io\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903670    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-cni-bin\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903717    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-etc-kubernetes\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:15.903871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903744    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903758    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-node-log\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903787    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903817    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdrrk\" (UniqueName: \"kubernetes.io/projected/eaaf6fa9-4605-446d-8948-77d89065d6e9-kube-api-access-rdrrk\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/434fae71-2d34-4e71-b384-83aa51cf0a28-host\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903870    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvwq5\" (UniqueName: \"kubernetes.io/projected/cc6804dc-8637-4596-8769-222cc116238e-kube-api-access-nvwq5\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-registration-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-sys-fs\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.903968    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47c85415-51eb-43b0-85d0-9ad397371ca2-host-slash\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:15.904793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.904014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a679bbf7-133d-425c-959d-8eb901ecdcf4-kubelet-config\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:15.906449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.906430    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:32:15.906638 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.906620    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 07 22:32:15.906862 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.906840    2582 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 07 22:32:15.907190 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.907040    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pjslb\""
Sep 07 22:32:15.926046 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.926017    2582 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-t65cc"
Sep 07 22:32:15.932224 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.931821    2582 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-t65cc"
Sep 07 22:32:15.992082 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:15.992045    2582 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 07 22:32:16.004297 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004257    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-socket-dir-parent\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.004502 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004407    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:16.004502 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.004427    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:16.004502 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004456    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-socket-dir-parent\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.004502 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-slash\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.004741 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.004512    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret podName:a679bbf7-133d-425c-959d-8eb901ecdcf4 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:16.504474843 +0000 UTC m=+2.970631818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret") pod "global-pull-secret-syncer-22swc" (UID: "a679bbf7-133d-425c-959d-8eb901ecdcf4") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:16.004741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovnkube-script-lib\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.004741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004644    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/434fae71-2d34-4e71-b384-83aa51cf0a28-serviceca\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:16.004741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004676    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysconfig\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.004741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004699    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysctl-conf\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.004741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004722    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-log-socket\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004745    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovn-node-metrics-cert\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-systemd\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004797    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-run\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-log-socket\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004826    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004511    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-slash\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovnkube-config\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004925    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-modprobe-d\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004953    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-device-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.004980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-systemd-units\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005033    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-tuned\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.005080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2710c9fe-7828-47a8-82ea-2dc09f39d7d9-agent-certs\") pod \"konnectivity-agent-cj55w\" (UID: \"2710c9fe-7828-47a8-82ea-2dc09f39d7d9\") " pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-system-cni-dir\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005119    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005150    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h27m2\" (UniqueName: \"kubernetes.io/projected/dbb33cfb-6569-441c-aaf6-045291b36801-kube-api-access-h27m2\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005177    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-system-cni-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005189    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/434fae71-2d34-4e71-b384-83aa51cf0a28-serviceca\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005203    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-kubelet\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005235    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-run-netns\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-cnibin\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005290    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-hosts-file\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005317    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-multus-certs\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005372    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysctl-d\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-lib-modules\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-tmp\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-tmp-dir\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:16.005816 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-etc-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005560    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-env-overrides\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005602    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-systemd-units\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nn574\" (UniqueName: \"kubernetes.io/projected/434fae71-2d34-4e71-b384-83aa51cf0a28-kube-api-access-nn574\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005669    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47c85415-51eb-43b0-85d0-9ad397371ca2-iptables-alerter-script\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-socket-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005718    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005742    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbrhh\" (UniqueName: \"kubernetes.io/projected/47c85415-51eb-43b0-85d0-9ad397371ca2-kube-api-access-cbrhh\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005747    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovnkube-script-lib\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-cni-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-run-netns\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-system-cni-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005864    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-kubelet-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005860    2582 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef39637-cc2e-48ec-866a-101b11cd6329-cni-binary-copy\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005922    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-etc-selinux\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.005977    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-hostroot\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006007    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef39637-cc2e-48ec-866a-101b11cd6329-multus-daemon-config\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.006743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006007    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-hosts-file\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-k8s-cni-cncf-io\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006061    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-cni-bin\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006087    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-etc-kubernetes\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-node-log\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006143    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006172    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-device-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006206    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-kubelet\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-cni-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006272    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdrrk\" (UniqueName: \"kubernetes.io/projected/eaaf6fa9-4605-446d-8948-77d89065d6e9-kube-api-access-rdrrk\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006299    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/434fae71-2d34-4e71-b384-83aa51cf0a28-host\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006349    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-sys\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006376    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvwq5\" (UniqueName: \"kubernetes.io/projected/cc6804dc-8637-4596-8769-222cc116238e-kube-api-access-nvwq5\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006403    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-registration-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006427    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-sys-fs\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47c85415-51eb-43b0-85d0-9ad397371ca2-host-slash\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006463    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/47c85415-51eb-43b0-85d0-9ad397371ca2-iptables-alerter-script\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:16.007744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006502    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovnkube-config\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006517    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a679bbf7-133d-425c-959d-8eb901ecdcf4-kubelet-config\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-tmp-dir\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006521    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-cni-bin\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006542    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-etc-kubernetes\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006556    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/47c85415-51eb-43b0-85d0-9ad397371ca2-host-slash\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006572    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-ovn\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006580    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-node-log\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006596    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-multus-certs\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-var-lib-kubelet\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006626    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-k8s-cni-cncf-io\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006652    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv56l\" (UniqueName: \"kubernetes.io/projected/cf3aa175-e534-4516-a436-d65670fc9198-kube-api-access-tv56l\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-registration-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006712    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-ovn\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006722    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-sys-fs\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a679bbf7-133d-425c-959d-8eb901ecdcf4-dbus\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006763    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.008553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006793    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2710c9fe-7828-47a8-82ea-2dc09f39d7d9-konnectivity-ca\") pod \"konnectivity-agent-cj55w\" (UID: \"2710c9fe-7828-47a8-82ea-2dc09f39d7d9\") " pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-etc-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006820    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eef39637-cc2e-48ec-866a-101b11cd6329-cni-binary-copy\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006831    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006853    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/a679bbf7-133d-425c-959d-8eb901ecdcf4-dbus\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006864    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006892    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/dbb33cfb-6569-441c-aaf6-045291b36801-socket-dir\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006895    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/a679bbf7-133d-425c-959d-8eb901ecdcf4-kubelet-config\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-hostroot\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006937    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-cnibin\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.006943    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/434fae71-2d34-4e71-b384-83aa51cf0a28-host\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.006988    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007018    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-cni-netd\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007060    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-cni-netd\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-cnibin\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007087    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-host\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.007265    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:16.507220769 +0000 UTC m=+2.973377760 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:16.009268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007287    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvhg5\" (UniqueName: \"kubernetes.io/projected/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-kube-api-access-wvhg5\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007287    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eef39637-cc2e-48ec-866a-101b11cd6329-multus-daemon-config\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007318    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-kubernetes\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007355    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-cni-bin\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007382    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/eaaf6fa9-4605-446d-8948-77d89065d6e9-env-overrides\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007404    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-cni-multus\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-cni-bin\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-netns\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007528    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-var-lib-cni-multus\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007581    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-host-run-netns\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85c6s\" (UniqueName: \"kubernetes.io/projected/eef39637-cc2e-48ec-866a-101b11cd6329-kube-api-access-85c6s\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-systemd\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-run-systemd\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007748    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007795    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-var-lib-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007799    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-run-ovn-kubernetes\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.010211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjnkl\" (UniqueName: \"kubernetes.io/projected/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-kube-api-access-mjnkl\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.011090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007859    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-var-lib-openvswitch\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.011090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007893    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.011090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-conf-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.011090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007964    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-kubelet\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.011090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.007968    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eef39637-cc2e-48ec-866a-101b11cd6329-multus-conf-dir\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.011090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.008063    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/eaaf6fa9-4605-446d-8948-77d89065d6e9-host-kubelet\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.012543 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.012409    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/eaaf6fa9-4605-446d-8948-77d89065d6e9-ovn-node-metrics-cert\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.014346 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.014299    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:16.014346 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.014321    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:16.014346 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.014337    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lbhzd for pod openshift-network-diagnostics/network-check-target-vrxqj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:16.014583 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.014414    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd podName:22f7133d-4d2a-4e79-acfc-44402fd0b947 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:16.514393174 +0000 UTC m=+2.980550149 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-lbhzd" (UniqueName: "kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd") pod "network-check-target-vrxqj" (UID: "22f7133d-4d2a-4e79-acfc-44402fd0b947") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:16.016554 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.016531    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvwq5\" (UniqueName: \"kubernetes.io/projected/cc6804dc-8637-4596-8769-222cc116238e-kube-api-access-nvwq5\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:16.017880 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.017858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvhg5\" (UniqueName: \"kubernetes.io/projected/6075fe40-91f5-4f32-ada2-e3a3ed7c91fc-kube-api-access-wvhg5\") pod \"node-resolver-62ddd\" (UID: \"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc\") " pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:16.018465 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.018443    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdrrk\" (UniqueName: \"kubernetes.io/projected/eaaf6fa9-4605-446d-8948-77d89065d6e9-kube-api-access-rdrrk\") pod \"ovnkube-node-h6c9s\" (UID: \"eaaf6fa9-4605-446d-8948-77d89065d6e9\") " pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.018556 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.018533    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85c6s\" (UniqueName: \"kubernetes.io/projected/eef39637-cc2e-48ec-866a-101b11cd6329-kube-api-access-85c6s\") pod \"multus-rjgxt\" (UID: \"eef39637-cc2e-48ec-866a-101b11cd6329\") " pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.019898 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.019878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nn574\" (UniqueName: \"kubernetes.io/projected/434fae71-2d34-4e71-b384-83aa51cf0a28-kube-api-access-nn574\") pod \"node-ca-fljbl\" (UID: \"434fae71-2d34-4e71-b384-83aa51cf0a28\") " pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:16.019996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.019953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbrhh\" (UniqueName: \"kubernetes.io/projected/47c85415-51eb-43b0-85d0-9ad397371ca2-kube-api-access-cbrhh\") pod \"iptables-alerter-gbd66\" (UID: \"47c85415-51eb-43b0-85d0-9ad397371ca2\") " pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:16.021890 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.021856    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h27m2\" (UniqueName: \"kubernetes.io/projected/dbb33cfb-6569-441c-aaf6-045291b36801-kube-api-access-h27m2\") pod \"aws-ebs-csi-driver-node-fcmmp\" (UID: \"dbb33cfb-6569-441c-aaf6-045291b36801\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.108351 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108291    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysconfig\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108351 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108361    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysctl-conf\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108392    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-systemd\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108415    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-run\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108444    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-modprobe-d\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-tuned\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108517    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-run\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2710c9fe-7828-47a8-82ea-2dc09f39d7d9-agent-certs\") pod \"konnectivity-agent-cj55w\" (UID: \"2710c9fe-7828-47a8-82ea-2dc09f39d7d9\") " pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108471    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysconfig\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108570    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-system-cni-dir\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-system-cni-dir\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108570    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-systemd\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108607    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.108657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-cnibin\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108664    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-modprobe-d\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108677    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysctl-d\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-lib-modules\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108741    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-tmp\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysctl-d\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108566    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-sysctl-conf\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108822    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-sys\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108855    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-var-lib-kubelet\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109275    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv56l\" (UniqueName: \"kubernetes.io/projected/cf3aa175-e534-4516-a436-d65670fc9198-kube-api-access-tv56l\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109343    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2710c9fe-7828-47a8-82ea-2dc09f39d7d9-konnectivity-ca\") pod \"konnectivity-agent-cj55w\" (UID: \"2710c9fe-7828-47a8-82ea-2dc09f39d7d9\") " pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-lib-modules\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109388    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109452    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-host\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109484    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-kubernetes\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109506    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-var-lib-kubelet\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109568    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjnkl\" (UniqueName: \"kubernetes.io/projected/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-kube-api-access-mjnkl\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.109719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109613    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.110181 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109862    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.110181 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.109967    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-cnibin\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.110181 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.108793    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cf3aa175-e534-4516-a436-d65670fc9198-tuning-conf-dir\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.110181 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.110062    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-host\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.110181 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.110127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-kubernetes\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.110486 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.110249    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-cni-binary-copy\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.110486 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.110340    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2710c9fe-7828-47a8-82ea-2dc09f39d7d9-konnectivity-ca\") pod \"konnectivity-agent-cj55w\" (UID: \"2710c9fe-7828-47a8-82ea-2dc09f39d7d9\") " pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:16.110486 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.110453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-sys\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.110650 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.110497    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/cf3aa175-e534-4516-a436-d65670fc9198-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.113741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.113705    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2710c9fe-7828-47a8-82ea-2dc09f39d7d9-agent-certs\") pod \"konnectivity-agent-cj55w\" (UID: \"2710c9fe-7828-47a8-82ea-2dc09f39d7d9\") " pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:16.114785 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.114759    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-etc-tuned\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.115306 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.115269    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-tmp\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.118384 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.118150    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv56l\" (UniqueName: \"kubernetes.io/projected/cf3aa175-e534-4516-a436-d65670fc9198-kube-api-access-tv56l\") pod \"multus-additional-cni-plugins-kx45q\" (UID: \"cf3aa175-e534-4516-a436-d65670fc9198\") " pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.118384 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.118282    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjnkl\" (UniqueName: \"kubernetes.io/projected/5c5a36f3-ec3a-4abe-bdc9-e965070a1436-kube-api-access-mjnkl\") pod \"tuned-pcvdb\" (UID: \"5c5a36f3-ec3a-4abe-bdc9-e965070a1436\") " pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.195781 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.195739    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp"
Sep 07 22:32:16.209194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.209161    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-gbd66"
Sep 07 22:32:16.217314 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.217277    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-62ddd"
Sep 07 22:32:16.219602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.219582    2582 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:16.225717 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.225693    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:16.234770 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.234745    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-fljbl"
Sep 07 22:32:16.243294 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.243264    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rjgxt"
Sep 07 22:32:16.252423 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.252395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:16.260481 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.260457    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-kx45q"
Sep 07 22:32:16.266490 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.266470    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pcvdb"
Sep 07 22:32:16.304682 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.304638    2582 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 07 22:32:16.512733 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.512482    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:16.512733 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.512707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:16.512943 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.512647    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:16.512943 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.512819    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:17.512801282 +0000 UTC m=+3.978958268 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:16.512943 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.512842    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:16.512943 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.512890    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret podName:a679bbf7-133d-425c-959d-8eb901ecdcf4 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:17.512873579 +0000 UTC m=+3.979030605 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret") pod "global-pull-secret-syncer-22swc" (UID: "a679bbf7-133d-425c-959d-8eb901ecdcf4") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:16.613143 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.613098    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:16.613358 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.613327    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:16.613358 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.613356    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:16.613496 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.613371    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lbhzd for pod openshift-network-diagnostics/network-check-target-vrxqj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:16.613496 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:16.613445    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd podName:22f7133d-4d2a-4e79-acfc-44402fd0b947 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:17.613423577 +0000 UTC m=+4.079580556 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbhzd" (UniqueName: "kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd") pod "network-check-target-vrxqj" (UID: "22f7133d-4d2a-4e79-acfc-44402fd0b947") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:16.933992 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.933876    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 22:27:16 +0000 UTC" deadline="2028-06-17 23:01:57.3446 +0000 UTC"
Sep 07 22:32:16.933992 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:16.933987    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15576h29m40.410621897s"
Sep 07 22:32:17.051755 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.051731    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47c85415_51eb_43b0_85d0_9ad397371ca2.slice/crio-5115a0eba339c14357e5ccd6aaadc3cf6c8b2f5e5331e9c3b02ace79999e339d WatchSource:0}: Error finding container 5115a0eba339c14357e5ccd6aaadc3cf6c8b2f5e5331e9c3b02ace79999e339d: Status 404 returned error can't find the container with id 5115a0eba339c14357e5ccd6aaadc3cf6c8b2f5e5331e9c3b02ace79999e339d
Sep 07 22:32:17.053144 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.053076    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6075fe40_91f5_4f32_ada2_e3a3ed7c91fc.slice/crio-9c5f84076ea6d0480697d4ef1fd31f9e74fb456e0f1a649842f2010fb8fa7c75 WatchSource:0}: Error finding container 9c5f84076ea6d0480697d4ef1fd31f9e74fb456e0f1a649842f2010fb8fa7c75: Status 404 returned error can't find the container with id 9c5f84076ea6d0480697d4ef1fd31f9e74fb456e0f1a649842f2010fb8fa7c75
Sep 07 22:32:17.054059 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.054000    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaaf6fa9_4605_446d_8948_77d89065d6e9.slice/crio-d85501d8adabd34bf60e61347ab6bcdc0365faab9eddccdcebc85f17fefbfdc2 WatchSource:0}: Error finding container d85501d8adabd34bf60e61347ab6bcdc0365faab9eddccdcebc85f17fefbfdc2: Status 404 returned error can't find the container with id d85501d8adabd34bf60e61347ab6bcdc0365faab9eddccdcebc85f17fefbfdc2
Sep 07 22:32:17.054824 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.054784    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef39637_cc2e_48ec_866a_101b11cd6329.slice/crio-59f620f7f3e6076a6e779f5c9868242f7c1814bfcac000c393c52c0deff2b179 WatchSource:0}: Error finding container 59f620f7f3e6076a6e779f5c9868242f7c1814bfcac000c393c52c0deff2b179: Status 404 returned error can't find the container with id 59f620f7f3e6076a6e779f5c9868242f7c1814bfcac000c393c52c0deff2b179
Sep 07 22:32:17.055855 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.055818    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2710c9fe_7828_47a8_82ea_2dc09f39d7d9.slice/crio-a7a686ffb4b31480a5f1226e819e4973cc471425678edb61c6c00802dfbb76a4 WatchSource:0}: Error finding container a7a686ffb4b31480a5f1226e819e4973cc471425678edb61c6c00802dfbb76a4: Status 404 returned error can't find the container with id a7a686ffb4b31480a5f1226e819e4973cc471425678edb61c6c00802dfbb76a4
Sep 07 22:32:17.059204 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.059082    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbb33cfb_6569_441c_aaf6_045291b36801.slice/crio-868310a4944d50a70e1a32ab0493759e74cc854473872265016c780802321039 WatchSource:0}: Error finding container 868310a4944d50a70e1a32ab0493759e74cc854473872265016c780802321039: Status 404 returned error can't find the container with id 868310a4944d50a70e1a32ab0493759e74cc854473872265016c780802321039
Sep 07 22:32:17.059916 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.059894    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod434fae71_2d34_4e71_b384_83aa51cf0a28.slice/crio-04032a9a1c1519f49c73377d9c5949cf0550b5412fe1e3c55e33494de8c6080a WatchSource:0}: Error finding container 04032a9a1c1519f49c73377d9c5949cf0550b5412fe1e3c55e33494de8c6080a: Status 404 returned error can't find the container with id 04032a9a1c1519f49c73377d9c5949cf0550b5412fe1e3c55e33494de8c6080a
Sep 07 22:32:17.061233 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.061212    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf3aa175_e534_4516_a436_d65670fc9198.slice/crio-5c870ea23f47dbaf30ae03357520637b773009f7582b347e36207ecda5709bbe WatchSource:0}: Error finding container 5c870ea23f47dbaf30ae03357520637b773009f7582b347e36207ecda5709bbe: Status 404 returned error can't find the container with id 5c870ea23f47dbaf30ae03357520637b773009f7582b347e36207ecda5709bbe
Sep 07 22:32:17.063491 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:17.062611    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c5a36f3_ec3a_4abe_bdc9_e965070a1436.slice/crio-20daaffacb0cf914bb3e64d3be10ee3f02da2f9c51b1dd5b4db65b7a05854096 WatchSource:0}: Error finding container 20daaffacb0cf914bb3e64d3be10ee3f02da2f9c51b1dd5b4db65b7a05854096: Status 404 returned error can't find the container with id 20daaffacb0cf914bb3e64d3be10ee3f02da2f9c51b1dd5b4db65b7a05854096
Sep 07 22:32:17.083734 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.083705    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:17.083871 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.083834    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:17.092243 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.092207    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fljbl" event={"ID":"434fae71-2d34-4e71-b384-83aa51cf0a28","Type":"ContainerStarted","Data":"04032a9a1c1519f49c73377d9c5949cf0550b5412fe1e3c55e33494de8c6080a"}
Sep 07 22:32:17.095261 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.095232    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp" event={"ID":"dbb33cfb-6569-441c-aaf6-045291b36801","Type":"ContainerStarted","Data":"868310a4944d50a70e1a32ab0493759e74cc854473872265016c780802321039"}
Sep 07 22:32:17.098917 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.098888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"d85501d8adabd34bf60e61347ab6bcdc0365faab9eddccdcebc85f17fefbfdc2"}
Sep 07 22:32:17.102656 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.102623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerStarted","Data":"5c870ea23f47dbaf30ae03357520637b773009f7582b347e36207ecda5709bbe"}
Sep 07 22:32:17.104728 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.104703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-62ddd" event={"ID":"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc","Type":"ContainerStarted","Data":"9c5f84076ea6d0480697d4ef1fd31f9e74fb456e0f1a649842f2010fb8fa7c75"}
Sep 07 22:32:17.105850 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.105827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cj55w" event={"ID":"2710c9fe-7828-47a8-82ea-2dc09f39d7d9","Type":"ContainerStarted","Data":"a7a686ffb4b31480a5f1226e819e4973cc471425678edb61c6c00802dfbb76a4"}
Sep 07 22:32:17.106975 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.106951    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rjgxt" event={"ID":"eef39637-cc2e-48ec-866a-101b11cd6329","Type":"ContainerStarted","Data":"59f620f7f3e6076a6e779f5c9868242f7c1814bfcac000c393c52c0deff2b179"}
Sep 07 22:32:17.108226 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.108200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbd66" event={"ID":"47c85415-51eb-43b0-85d0-9ad397371ca2","Type":"ContainerStarted","Data":"5115a0eba339c14357e5ccd6aaadc3cf6c8b2f5e5331e9c3b02ace79999e339d"}
Sep 07 22:32:17.109384 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.109364    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pcvdb" event={"ID":"5c5a36f3-ec3a-4abe-bdc9-e965070a1436","Type":"ContainerStarted","Data":"20daaffacb0cf914bb3e64d3be10ee3f02da2f9c51b1dd5b4db65b7a05854096"}
Sep 07 22:32:17.521206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.521107    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:17.521206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.521188    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:17.521484 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.521287    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:17.521484 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.521387    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:19.521364092 +0000 UTC m=+5.987521089 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:17.521902 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.521307    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:17.521902 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.521828    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret podName:a679bbf7-133d-425c-959d-8eb901ecdcf4 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:19.521810325 +0000 UTC m=+5.987967327 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret") pod "global-pull-secret-syncer-22swc" (UID: "a679bbf7-133d-425c-959d-8eb901ecdcf4") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:17.622351 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.622292    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:17.622575 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.622562    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:17.622640 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.622586    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:17.622640 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.622601    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lbhzd for pod openshift-network-diagnostics/network-check-target-vrxqj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:17.622748 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:17.622664    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd podName:22f7133d-4d2a-4e79-acfc-44402fd0b947 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:19.622645505 +0000 UTC m=+6.088802485 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbhzd" (UniqueName: "kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd") pod "network-check-target-vrxqj" (UID: "22f7133d-4d2a-4e79-acfc-44402fd0b947") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:17.934553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.934422    2582 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-06 22:27:16 +0000 UTC" deadline="2028-03-10 04:18:38.18622034 +0000 UTC"
Sep 07 22:32:17.934553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:17.934493    2582 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13181h46m20.251731431s"
Sep 07 22:32:18.085423 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:18.085384    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:18.085641 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:18.085539    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:18.087696 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:18.087422    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:18.087696 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:18.087551    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:18.124191 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:18.124108    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal" event={"ID":"bbd0e3824c495bf54791857c054ed650","Type":"ContainerStarted","Data":"42f2068ac47a1aeb3c0e141b32c355fe2037464bf9b03f6330d79addbb7ce737"}
Sep 07 22:32:18.127717 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:18.127685    2582 generic.go:356] "Generic (PLEG): container finished" podID="5625e0c08c7af636ceb9734794ed4c2f" containerID="f69c038ad1f5eed2d680bae08ec1f7200a7808d0e986c554db1b6d5e4945198e" exitCode=0
Sep 07 22:32:18.127866 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:18.127739    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal" event={"ID":"5625e0c08c7af636ceb9734794ed4c2f","Type":"ContainerDied","Data":"f69c038ad1f5eed2d680bae08ec1f7200a7808d0e986c554db1b6d5e4945198e"}
Sep 07 22:32:18.139319 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:18.139261    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-80.ec2.internal" podStartSLOduration=3.1392390470000002 podStartE2EDuration="3.139239047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:18.138500314 +0000 UTC m=+4.604657314" watchObservedRunningTime="2026-09-07 22:32:18.139239047 +0000 UTC m=+4.605396048"
Sep 07 22:32:19.083453 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:19.083412    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:19.084117 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.083563    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:19.146528 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:19.146485    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal" event={"ID":"5625e0c08c7af636ceb9734794ed4c2f","Type":"ContainerStarted","Data":"30ee67fa5218b44e8854ca6a56b1559307c90e90b4a40e4e8cbf15ddc531b81a"}
Sep 07 22:32:19.539209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:19.539115    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:19.539209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:19.539199    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:19.539484 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.539458    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:19.539549 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.539529    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:23.539508627 +0000 UTC m=+10.005665609 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:19.539932 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.539910    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:19.540019 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.539987    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret podName:a679bbf7-133d-425c-959d-8eb901ecdcf4 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:23.539958886 +0000 UTC m=+10.006115880 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret") pod "global-pull-secret-syncer-22swc" (UID: "a679bbf7-133d-425c-959d-8eb901ecdcf4") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:19.640199 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:19.640155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:19.640421 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.640391    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:19.640421 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.640415    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:19.640536 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.640429    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lbhzd for pod openshift-network-diagnostics/network-check-target-vrxqj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:19.640536 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:19.640494    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd podName:22f7133d-4d2a-4e79-acfc-44402fd0b947 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:23.640473935 +0000 UTC m=+10.106630912 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbhzd" (UniqueName: "kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd") pod "network-check-target-vrxqj" (UID: "22f7133d-4d2a-4e79-acfc-44402fd0b947") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:20.084516 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:20.084473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:20.084516 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:20.084491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:20.085069 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:20.084639    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:20.085069 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:20.084799    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:21.083859 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:21.083453    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:21.083859 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:21.083586    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:22.084933 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:22.084411    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:22.084933 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:22.084432    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:22.084933 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:22.084561    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:22.084933 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:22.084654    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:23.083572 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:23.083537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:23.083758 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.083691    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:23.574699 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:23.574430    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:23.574699 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:23.574524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:23.574699 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.574597    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:23.574699 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.574618    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:23.574699 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.574668    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret podName:a679bbf7-133d-425c-959d-8eb901ecdcf4 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:31.574650413 +0000 UTC m=+18.040807390 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret") pod "global-pull-secret-syncer-22swc" (UID: "a679bbf7-133d-425c-959d-8eb901ecdcf4") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:23.574699 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.574690    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:31.57467913 +0000 UTC m=+18.040836112 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:23.675753 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:23.675715    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:23.675938 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.675899    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:23.675938 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.675925    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:23.675938 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.675937    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lbhzd for pod openshift-network-diagnostics/network-check-target-vrxqj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:23.676097 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:23.676005    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd podName:22f7133d-4d2a-4e79-acfc-44402fd0b947 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:31.675984599 +0000 UTC m=+18.142141588 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbhzd" (UniqueName: "kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd") pod "network-check-target-vrxqj" (UID: "22f7133d-4d2a-4e79-acfc-44402fd0b947") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:24.086313 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:24.085910    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:24.086313 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:24.086045    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:24.086559 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:24.086434    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:24.086559 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:24.086529    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:25.084345 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:25.084307    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:25.084815 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:25.084451    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:26.083763 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:26.083670    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:26.083955 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:26.083813    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:26.083955 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:26.083863    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:26.084070 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:26.083971    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:27.084023 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:27.083987    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:27.084435 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:27.084103    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:28.083801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:28.083575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:28.083987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:28.083627    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:28.083987 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:28.083906    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:28.084441 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:28.084011    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:29.084395 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:29.084358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:29.084848 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:29.084489    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:30.084250 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:30.084210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:30.084416 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:30.084367    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:30.084770 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:30.084210    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:30.084897 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:30.084874    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:31.084254 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:31.084216    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:31.084457 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.084343    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:31.633580 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:31.633527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:31.633741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:31.633616    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:31.633741 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.633697    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:31.633741 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.633714    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:31.633873 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.633766    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret podName:a679bbf7-133d-425c-959d-8eb901ecdcf4 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:47.633752325 +0000 UTC m=+34.099909302 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret") pod "global-pull-secret-syncer-22swc" (UID: "a679bbf7-133d-425c-959d-8eb901ecdcf4") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:31.633873 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.633782    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:32:47.633775571 +0000 UTC m=+34.099932544 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:31.734237 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:31.734202    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:31.734427 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.734345    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:31.734427 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.734366    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:31.734427 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.734379    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lbhzd for pod openshift-network-diagnostics/network-check-target-vrxqj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:31.734551 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:31.734441    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd podName:22f7133d-4d2a-4e79-acfc-44402fd0b947 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:47.73442767 +0000 UTC m=+34.200584661 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbhzd" (UniqueName: "kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd") pod "network-check-target-vrxqj" (UID: "22f7133d-4d2a-4e79-acfc-44402fd0b947") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:32.083500 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:32.083462    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:32.083710 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:32.083459    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:32.083710 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:32.083597    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:32.083710 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:32.083649    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:33.083567 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:33.083531    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:33.084006 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:33.083663    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:34.084885 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:34.084847    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:34.085340 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:34.084975    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:34.085340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:34.085000    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:34.085340 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:34.085125    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:35.085081 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.084326    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:35.085081 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:35.084599    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:35.184019 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.183976    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf3aa175-e534-4516-a436-d65670fc9198" containerID="4c917ea7a5e4c42396c3423ee4736660faab4e186d6f4d9d21195320cf8389e0" exitCode=0
Sep 07 22:32:35.184152 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.184054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerDied","Data":"4c917ea7a5e4c42396c3423ee4736660faab4e186d6f4d9d21195320cf8389e0"}
Sep 07 22:32:35.185909 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.185881    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-62ddd" event={"ID":"6075fe40-91f5-4f32-ada2-e3a3ed7c91fc","Type":"ContainerStarted","Data":"d8a0ce2af97d91abb88d7a79d3b7fe5615c07955381765e22af0b78d1ab9266a"}
Sep 07 22:32:35.187950 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.187918    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cj55w" event={"ID":"2710c9fe-7828-47a8-82ea-2dc09f39d7d9","Type":"ContainerStarted","Data":"a11a77132d957f84da570921f2de61528c673ac17ec60f0f10451c9309e418f6"}
Sep 07 22:32:35.189918 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.189857    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rjgxt" event={"ID":"eef39637-cc2e-48ec-866a-101b11cd6329","Type":"ContainerStarted","Data":"dd068cddf84b09438ae51d4cb9cd794e914fff1034d7fcabdb235a917facf334"}
Sep 07 22:32:35.191563 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.191538    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pcvdb" event={"ID":"5c5a36f3-ec3a-4abe-bdc9-e965070a1436","Type":"ContainerStarted","Data":"478515f4ad092d21335cd5a698802494b5bd1a73cb0958d88a0500e024f5b744"}
Sep 07 22:32:35.192922 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.192904    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-fljbl" event={"ID":"434fae71-2d34-4e71-b384-83aa51cf0a28","Type":"ContainerStarted","Data":"2ed5276792e4081c6860a73b8b5e145d502449586ccc15e78f9d49c7bd09d14d"}
Sep 07 22:32:35.194416 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.194396    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp" event={"ID":"dbb33cfb-6569-441c-aaf6-045291b36801","Type":"ContainerStarted","Data":"5ad14e9e2f534972afd2819c1c8a093ab01e94565439f2c29fcbbcb98780afad"}
Sep 07 22:32:35.196129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.196111    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"ee52b514bfb0debc0a1108c74e94c68f7bb81ea1a8ad655753fd32ecaf0f1517"}
Sep 07 22:32:35.196204 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.196168    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"6fbbff4a007a3c7c5bbe54bffd8a07770720ec2203ef863345549dd7aebd9eae"}
Sep 07 22:32:35.203641 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.203599    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-80.ec2.internal" podStartSLOduration=20.203584097 podStartE2EDuration="20.203584097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:19.16122334 +0000 UTC m=+5.627380340" watchObservedRunningTime="2026-09-07 22:32:35.203584097 +0000 UTC m=+21.669741092"
Sep 07 22:32:35.233424 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.233239    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-rjgxt" podStartSLOduration=3.400401655 podStartE2EDuration="21.233216076s" totalImagesPullingTime="17.832814421s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:35.218701535 +0000 UTC m=+21.684858552" watchObservedRunningTime="2026-09-07 22:32:35.233216076 +0000 UTC m=+21.699373071"
Sep 07 22:32:35.234214 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.233917    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cj55w" podStartSLOduration=7.268134013 podStartE2EDuration="21.233890782s" totalImagesPullingTime="13.965756769s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:35.232992259 +0000 UTC m=+21.699149253" watchObservedRunningTime="2026-09-07 22:32:35.233890782 +0000 UTC m=+21.700047779"
Sep 07 22:32:35.247618 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.247569    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pcvdb" podStartSLOduration=3.46351844 podStartE2EDuration="21.24755533s" totalImagesPullingTime="17.78403689s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:35.247494956 +0000 UTC m=+21.713651952" watchObservedRunningTime="2026-09-07 22:32:35.24755533 +0000 UTC m=+21.713712324"
Sep 07 22:32:35.261651 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.261608    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-62ddd" podStartSLOduration=3.485661051 podStartE2EDuration="21.261593378s" totalImagesPullingTime="17.775932327s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:35.261240077 +0000 UTC m=+21.727397072" watchObservedRunningTime="2026-09-07 22:32:35.261593378 +0000 UTC m=+21.727750373"
Sep 07 22:32:35.276296 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:35.276241    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-fljbl" podStartSLOduration=3.506653745 podStartE2EDuration="21.27622555s" totalImagesPullingTime="17.769571805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:35.275745539 +0000 UTC m=+21.741902535" watchObservedRunningTime="2026-09-07 22:32:35.27622555 +0000 UTC m=+21.742382545"
Sep 07 22:32:36.083434 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:36.083392    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:36.083642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:36.083393    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:36.083642 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:36.083525    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:36.083642 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:36.083608    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:36.202365 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:36.202266    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"616b83ea746c03f6fa9f979649fb6a84f254be3e58b09dd105e491320c088c39"}
Sep 07 22:32:36.203144 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:36.202383    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"ed95a707a009cbd0ac1e022b8d8ec7c76df7346b9c945662a5f9dc2c3e26d416"}
Sep 07 22:32:36.203144 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:36.202434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"db8522fe5196173d76969640430e326fb602d8b0d6c487abbcd59f9a844f7bb5"}
Sep 07 22:32:36.203144 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:36.202481    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"4498e642face448e9575d259664fd52d0b554a757788ade7b49ee6770197c8eb"}
Sep 07 22:32:36.518636 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:36.518451    2582 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 07 22:32:37.083650 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.083616    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:37.083942 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:37.083752    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:37.098131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.098021    2582 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-07T22:32:36.518613815Z","UUID":"053d9b9c-1fab-4668-ab13-2afd19f78a00","Handler":null,"Name":"","Endpoint":""}
Sep 07 22:32:37.099802 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.099780    2582 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 07 22:32:37.099923 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.099815    2582 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 07 22:32:37.207320 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.207092    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-gbd66" event={"ID":"47c85415-51eb-43b0-85d0-9ad397371ca2","Type":"ContainerStarted","Data":"7a2a8d0c6cd482d8e4b1f403d4c60d9f4423fdce37207b372d9297e5403ee7a1"}
Sep 07 22:32:37.209535 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.209504    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp" event={"ID":"dbb33cfb-6569-441c-aaf6-045291b36801","Type":"ContainerStarted","Data":"74cf77cf05dfbea2b5fbf95dd3193d347710d96bb0918f5626d28ba719f61dac"}
Sep 07 22:32:37.586419 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.586195    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:37.587134 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.587018    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:37.600510 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:37.600456    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-gbd66" podStartSLOduration=5.817668578 podStartE2EDuration="23.600432448s" totalImagesPullingTime="17.78276387s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:37.220260273 +0000 UTC m=+23.686417267" watchObservedRunningTime="2026-09-07 22:32:37.600432448 +0000 UTC m=+24.066589444"
Sep 07 22:32:38.083362 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:38.083334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:38.083538 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:38.083334    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:38.083538 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:38.083444    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:38.083641 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:38.083563    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:38.214904 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:38.214862    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp" event={"ID":"dbb33cfb-6569-441c-aaf6-045291b36801","Type":"ContainerStarted","Data":"f7ea566dd092d6a245564a92c700b8455d16540a4c6b9a3cd7ee3168c9940cc0"}
Sep 07 22:32:38.222060 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:38.222014    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"95a8c812422d3ef227da2dadc402a7f3df0aab8284e61760a5a096af8e189603"}
Sep 07 22:32:38.222356 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:38.222326    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:38.222856 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:38.222838    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cj55w"
Sep 07 22:32:38.232487 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:38.232435    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-fcmmp" podStartSLOduration=3.90221557 podStartE2EDuration="24.23241529s" totalImagesPullingTime="20.33019972s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:38.231979203 +0000 UTC m=+24.698136200" watchObservedRunningTime="2026-09-07 22:32:38.23241529 +0000 UTC m=+24.698572291"
Sep 07 22:32:39.083766 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:39.083726    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:39.083969 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:39.083868    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:40.083629 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:40.083452    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:40.084052 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:40.083515    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:40.084052 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:40.083779    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:40.084052 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:40.083877    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:40.228589 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:40.228509    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf3aa175-e534-4516-a436-d65670fc9198" containerID="8bcd42c838bffd127d47db9cd21cda6ac89488502ed75e601966a0add45c980b" exitCode=0
Sep 07 22:32:40.228937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:40.228596    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerDied","Data":"8bcd42c838bffd127d47db9cd21cda6ac89488502ed75e601966a0add45c980b"}
Sep 07 22:32:40.232593 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:40.232465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" event={"ID":"eaaf6fa9-4605-446d-8948-77d89065d6e9","Type":"ContainerStarted","Data":"aeae092e627f5d190047147f539cae8f423a3e8d70ff82496d30a9b6c801145e"}
Sep 07 22:32:40.270896 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:40.270842    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s" podStartSLOduration=8.442320687 podStartE2EDuration="26.270825877s" totalImagesPullingTime="17.82850519s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:40.269153505 +0000 UTC m=+26.735310502" watchObservedRunningTime="2026-09-07 22:32:40.270825877 +0000 UTC m=+26.736982872"
Sep 07 22:32:40.489769 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:40.489682    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:41.083585 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.083537    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:41.083815 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:41.083676    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:41.235514 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.235401    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:41.235514 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.235492    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:41.254609 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.254576    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:41.255117 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.255092    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:32:41.986139 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.986096    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vrxqj"]
Sep 07 22:32:41.986341 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.986212    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:41.986415 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:41.986336    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:41.986792 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.986774    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-22swc"]
Sep 07 22:32:41.986894 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.986858    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:41.986942 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:41.986926    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:41.996855 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.996788    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sqns"]
Sep 07 22:32:41.996981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:41.996906    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:41.997042 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:41.997023    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:42.239208 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:42.239170    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf3aa175-e534-4516-a436-d65670fc9198" containerID="ace6fe69435a004138bc849c05dca3fe3a541de2bbe43fa9e4daa037cc67682f" exitCode=0
Sep 07 22:32:42.239902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:42.239245    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerDied","Data":"ace6fe69435a004138bc849c05dca3fe3a541de2bbe43fa9e4daa037cc67682f"}
Sep 07 22:32:44.084592 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:44.084417    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:44.084981 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:44.084652    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:44.084981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:44.084487    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:44.084981 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:44.084724    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:44.084981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:44.084466    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:44.084981 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:44.084795    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:44.246698 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:44.246666    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf3aa175-e534-4516-a436-d65670fc9198" containerID="1bdfc079a4fd05d8e2d4096f3a14109efe58cf3472c22d80a5fc5dacbf8fa65b" exitCode=0
Sep 07 22:32:44.246874 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:44.246713    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerDied","Data":"1bdfc079a4fd05d8e2d4096f3a14109efe58cf3472c22d80a5fc5dacbf8fa65b"}
Sep 07 22:32:46.084314 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:46.084267    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:46.084783 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:46.084268    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:46.084783 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:46.084391    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-22swc" podUID="a679bbf7-133d-425c-959d-8eb901ecdcf4"
Sep 07 22:32:46.084783 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:46.084268    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:46.084783 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:46.084464    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-vrxqj" podUID="22f7133d-4d2a-4e79-acfc-44402fd0b947"
Sep 07 22:32:46.084783 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:46.084574    2582 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-4sqns" podUID="cc6804dc-8637-4596-8769-222cc116238e"
Sep 07 22:32:47.646610 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.646511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:47.646610 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.646577    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:47.647047 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.646675    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:47.647047 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.646683    2582 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:47.647047 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.646749    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:33:19.646728815 +0000 UTC m=+66.112885816 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 07 22:32:47.647047 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.646767    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret podName:a679bbf7-133d-425c-959d-8eb901ecdcf4 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:19.646760035 +0000 UTC m=+66.112917012 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret") pod "global-pull-secret-syncer-22swc" (UID: "a679bbf7-133d-425c-959d-8eb901ecdcf4") : object "kube-system"/"original-pull-secret" not registered
Sep 07 22:32:47.747588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.747550    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:47.747746 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.747722    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 07 22:32:47.747746 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.747746    2582 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 07 22:32:47.747821 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.747759    2582 projected.go:196] Error preparing data for projected volume kube-api-access-lbhzd for pod openshift-network-diagnostics/network-check-target-vrxqj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:47.747861 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:47.747821    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd podName:22f7133d-4d2a-4e79-acfc-44402fd0b947 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:19.747803271 +0000 UTC m=+66.213960253 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-lbhzd" (UniqueName: "kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd") pod "network-check-target-vrxqj" (UID: "22f7133d-4d2a-4e79-acfc-44402fd0b947") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 07 22:32:47.847625 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.847589    2582 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-80.ec2.internal" event="NodeReady"
Sep 07 22:32:47.847809 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.847742    2582 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 07 22:32:47.887609 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.887577    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-tthjd"]
Sep 07 22:32:47.892340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.892306    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-nr7g8"]
Sep 07 22:32:47.892505 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.892442    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:47.894753 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.894721    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zd2nk\""
Sep 07 22:32:47.894753 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.894742    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 07 22:32:47.894930 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.894721    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 07 22:32:47.895395 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.895368    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:47.897425 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.897371    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-zz6fl\""
Sep 07 22:32:47.897425 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.897416    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 07 22:32:47.897572 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.897542    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 07 22:32:47.897680 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.897584    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 07 22:32:47.899078 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.899060    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tthjd"]
Sep 07 22:32:47.902495 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.902066    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nr7g8"]
Sep 07 22:32:47.948720 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.948679    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:47.948927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.948730    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zqc8\" (UniqueName: \"kubernetes.io/projected/48ba9096-4eaf-4076-bba7-4e6cecd4d23c-kube-api-access-5zqc8\") pod \"ingress-canary-nr7g8\" (UID: \"48ba9096-4eaf-4076-bba7-4e6cecd4d23c\") " pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:47.948927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.948812    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgh4l\" (UniqueName: \"kubernetes.io/projected/12799779-c22a-4ead-829f-e75c7eff4644-kube-api-access-kgh4l\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:47.948927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.948854    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48ba9096-4eaf-4076-bba7-4e6cecd4d23c-cert\") pod \"ingress-canary-nr7g8\" (UID: \"48ba9096-4eaf-4076-bba7-4e6cecd4d23c\") " pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:47.949074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.948916    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12799779-c22a-4ead-829f-e75c7eff4644-tmp-dir\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:47.949074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:47.948992    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12799779-c22a-4ead-829f-e75c7eff4644-config-volume\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.049773 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.049730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.049773 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.049777    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zqc8\" (UniqueName: \"kubernetes.io/projected/48ba9096-4eaf-4076-bba7-4e6cecd4d23c-kube-api-access-5zqc8\") pod \"ingress-canary-nr7g8\" (UID: \"48ba9096-4eaf-4076-bba7-4e6cecd4d23c\") " pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:48.050019 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.049811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgh4l\" (UniqueName: \"kubernetes.io/projected/12799779-c22a-4ead-829f-e75c7eff4644-kube-api-access-kgh4l\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.050019 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.049828    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48ba9096-4eaf-4076-bba7-4e6cecd4d23c-cert\") pod \"ingress-canary-nr7g8\" (UID: \"48ba9096-4eaf-4076-bba7-4e6cecd4d23c\") " pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:48.050019 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.049877    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12799779-c22a-4ead-829f-e75c7eff4644-tmp-dir\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.050019 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.049902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12799779-c22a-4ead-829f-e75c7eff4644-config-volume\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.050019 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:48.049918    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:48.050019 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:48.049987    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls podName:12799779-c22a-4ead-829f-e75c7eff4644 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:48.549966969 +0000 UTC m=+35.016123960 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls") pod "dns-default-tthjd" (UID: "12799779-c22a-4ead-829f-e75c7eff4644") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:48.050372 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.050352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12799779-c22a-4ead-829f-e75c7eff4644-tmp-dir\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.050569 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.050551    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12799779-c22a-4ead-829f-e75c7eff4644-config-volume\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.054694 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.054673    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/48ba9096-4eaf-4076-bba7-4e6cecd4d23c-cert\") pod \"ingress-canary-nr7g8\" (UID: \"48ba9096-4eaf-4076-bba7-4e6cecd4d23c\") " pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:48.059546 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.059516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgh4l\" (UniqueName: \"kubernetes.io/projected/12799779-c22a-4ead-829f-e75c7eff4644-kube-api-access-kgh4l\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.059656 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.059593    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zqc8\" (UniqueName: \"kubernetes.io/projected/48ba9096-4eaf-4076-bba7-4e6cecd4d23c-kube-api-access-5zqc8\") pod \"ingress-canary-nr7g8\" (UID: \"48ba9096-4eaf-4076-bba7-4e6cecd4d23c\") " pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:48.083541 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.083508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:32:48.083703 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.083549    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:32:48.083769 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.083721    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:32:48.086664 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.086505    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lg5n9\""
Sep 07 22:32:48.086664 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.086564    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 22:32:48.086664 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.086608    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 22:32:48.086900 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.086791    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qh6q\""
Sep 07 22:32:48.087124 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.087100    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 22:32:48.087461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.087441    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 22:32:48.213436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.213354    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-nr7g8"
Sep 07 22:32:48.554080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:48.553993    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:48.554245 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:48.554175    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:48.554327 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:48.554260    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls podName:12799779-c22a-4ead-829f-e75c7eff4644 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:49.554237251 +0000 UTC m=+36.020394237 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls") pod "dns-default-tthjd" (UID: "12799779-c22a-4ead-829f-e75c7eff4644") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:49.012561 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:49.012322    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-nr7g8"]
Sep 07 22:32:49.041102 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:32:49.041074    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ba9096_4eaf_4076_bba7_4e6cecd4d23c.slice/crio-788f8709ed2055a99e98c6e87a7f4ccbe989c698874e98445364b910a06e96c0 WatchSource:0}: Error finding container 788f8709ed2055a99e98c6e87a7f4ccbe989c698874e98445364b910a06e96c0: Status 404 returned error can't find the container with id 788f8709ed2055a99e98c6e87a7f4ccbe989c698874e98445364b910a06e96c0
Sep 07 22:32:49.259581 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:49.259488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nr7g8" event={"ID":"48ba9096-4eaf-4076-bba7-4e6cecd4d23c","Type":"ContainerStarted","Data":"788f8709ed2055a99e98c6e87a7f4ccbe989c698874e98445364b910a06e96c0"}
Sep 07 22:32:49.262406 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:49.262379    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf3aa175-e534-4516-a436-d65670fc9198" containerID="85cf7ec67cd7b1b466d19ea9c6f8d78663e1f5e925a0a47311aad50c14ae0fa2" exitCode=0
Sep 07 22:32:49.262543 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:49.262438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerDied","Data":"85cf7ec67cd7b1b466d19ea9c6f8d78663e1f5e925a0a47311aad50c14ae0fa2"}
Sep 07 22:32:49.562533 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:49.562490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:49.562726 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:49.562665    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:49.562800 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:49.562750    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls podName:12799779-c22a-4ead-829f-e75c7eff4644 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:51.562728452 +0000 UTC m=+38.028885426 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls") pod "dns-default-tthjd" (UID: "12799779-c22a-4ead-829f-e75c7eff4644") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:50.269556 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:50.269511    2582 generic.go:356] "Generic (PLEG): container finished" podID="cf3aa175-e534-4516-a436-d65670fc9198" containerID="d0d52011258aa1bd3b192b83d68f2fddc1ba4e29c41369a0aaa507eb898d18c3" exitCode=0
Sep 07 22:32:50.269556 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:50.269546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerDied","Data":"d0d52011258aa1bd3b192b83d68f2fddc1ba4e29c41369a0aaa507eb898d18c3"}
Sep 07 22:32:51.277875 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:51.277837    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-kx45q" event={"ID":"cf3aa175-e534-4516-a436-d65670fc9198","Type":"ContainerStarted","Data":"cb5e7d926b3048fbf0feb47e3ee2e99cf4c556b093dd493d327a803396cf9704"}
Sep 07 22:32:51.297465 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:51.297418    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-kx45q" podStartSLOduration=8.349338568 podStartE2EDuration="37.297399524s" totalImagesPullingTime="28.948060956s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:51.295957323 +0000 UTC m=+37.762114317" watchObservedRunningTime="2026-09-07 22:32:51.297399524 +0000 UTC m=+37.763556520"
Sep 07 22:32:51.578245 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:51.578008    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:51.578435 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:51.578173    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:51.578435 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:51.578371    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls podName:12799779-c22a-4ead-829f-e75c7eff4644 nodeName:}" failed. No retries permitted until 2026-09-07 22:32:55.578355373 +0000 UTC m=+42.044512365 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls") pod "dns-default-tthjd" (UID: "12799779-c22a-4ead-829f-e75c7eff4644") : secret "dns-default-metrics-tls" not found
Sep 07 22:32:52.282063 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:52.282020    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-nr7g8" event={"ID":"48ba9096-4eaf-4076-bba7-4e6cecd4d23c","Type":"ContainerStarted","Data":"a841978fa52542451dc1c16ff00c2f0d203420434b4dc83a286ed6257c25cfdd"}
Sep 07 22:32:52.296221 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:52.296164    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-nr7g8" podStartSLOduration=2.873380319 podStartE2EDuration="5.296147293s" totalImagesPullingTime="2.422766974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:32:52.295209902 +0000 UTC m=+38.761366896" watchObservedRunningTime="2026-09-07 22:32:52.296147293 +0000 UTC m=+38.762304315"
Sep 07 22:32:55.604419 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:32:55.604370    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:32:55.604825 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:55.604527    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:32:55.604825 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:32:55.604599    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls podName:12799779-c22a-4ead-829f-e75c7eff4644 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:03.604580117 +0000 UTC m=+50.070737107 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls") pod "dns-default-tthjd" (UID: "12799779-c22a-4ead-829f-e75c7eff4644") : secret "dns-default-metrics-tls" not found
Sep 07 22:33:03.666061 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:03.665479    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:33:03.666061 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:03.665597    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls podName:12799779-c22a-4ead-829f-e75c7eff4644 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:19.665572538 +0000 UTC m=+66.131729528 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls") pod "dns-default-tthjd" (UID: "12799779-c22a-4ead-829f-e75c7eff4644") : secret "dns-default-metrics-tls" not found
Sep 07 22:33:03.668324 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:03.666324    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:33:07.894447 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.894411    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"]
Sep 07 22:33:07.896135 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.896120    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:07.898519 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.898496    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Sep 07 22:33:07.898784 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.898763    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Sep 07 22:33:07.899675 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.899653    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Sep 07 22:33:07.899675 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.899674    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:33:07.899842 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.899769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-ckkm7\""
Sep 07 22:33:07.907921 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.907894    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"]
Sep 07 22:33:07.995415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.995381    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-serving-cert\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:07.995415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.995420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-config\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:07.995626 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.995444    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7prt\" (UniqueName: \"kubernetes.io/projected/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-kube-api-access-k7prt\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:07.995912 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.995891    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6cf8f549f5-6r27h"]
Sep 07 22:33:07.997773 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.997758    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:07.999980 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:07.999942    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 07 22:33:08.000223 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.000208    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 07 22:33:08.000324 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.000275    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 07 22:33:08.000751 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.000733    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvdml\""
Sep 07 22:33:08.005985 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.005964    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 07 22:33:08.008895 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.008870    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cf8f549f5-6r27h"]
Sep 07 22:33:08.086877 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.086847    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-mkfjw"]
Sep 07 22:33:08.088731 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.088709    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.091097 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.091067    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Sep 07 22:33:08.091097 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.091095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Sep 07 22:33:08.091259 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.091173    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-l6dtc\""
Sep 07 22:33:08.091428 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.091415    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 07 22:33:08.091784 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.091765    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 07 22:33:08.095776 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.095753    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-installation-pull-secrets\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.095887 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.095796    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e368b88-1ea7-45f5-87a1-f83802318365-ca-trust-extracted\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.095958 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.095892    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-bound-sa-token\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.096010 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.095967    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-serving-cert\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:08.096063 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096021    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.096118 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-image-registry-private-configuration\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.096118 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096101    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-trusted-ca\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.096214 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096145    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-config\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:08.096264 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096209    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-registry-certificates\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.096339 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096260    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7prt\" (UniqueName: \"kubernetes.io/projected/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-kube-api-access-k7prt\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:08.096412 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz582\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-kube-api-access-jz582\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.096761 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-config\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:08.096996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.096979    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Sep 07 22:33:08.099203 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.099175    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-mkfjw"]
Sep 07 22:33:08.099918 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.099898    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-serving-cert\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:08.107072 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.107050    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7prt\" (UniqueName: \"kubernetes.io/projected/76d268f3-0e43-4b16-8386-cd3c4e9a6ee8-kube-api-access-k7prt\") pod \"service-ca-operator-859f94c4b-l9v78\" (UID: \"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8\") " pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:08.197345 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197217    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-installation-pull-secrets\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.197345 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197261    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e368b88-1ea7-45f5-87a1-f83802318365-ca-trust-extracted\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.197345 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197308    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-bound-sa-token\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.197345 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cd7b3adc-224b-4509-a86c-a892d692883d-snapshots\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.197345 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197344    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7b3adc-224b-4509-a86c-a892d692883d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197359    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd7b3adc-224b-4509-a86c-a892d692883d-serving-cert\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197384    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7b3adc-224b-4509-a86c-a892d692883d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197483    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:08.197552    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:08.197568    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf8f549f5-6r27h: secret "image-registry-tls" not found
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:08.197624    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls podName:4e368b88-1ea7-45f5-87a1-f83802318365 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:08.697604567 +0000 UTC m=+55.163761547 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls") pod "image-registry-6cf8f549f5-6r27h" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365") : secret "image-registry-tls" not found
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197551    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-image-registry-private-configuration\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.197686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-trusted-ca\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.198074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jm7ls\" (UniqueName: \"kubernetes.io/projected/cd7b3adc-224b-4509-a86c-a892d692883d-kube-api-access-jm7ls\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.198074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-registry-certificates\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.198074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd7b3adc-224b-4509-a86c-a892d692883d-tmp\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.198074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197788    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e368b88-1ea7-45f5-87a1-f83802318365-ca-trust-extracted\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.198074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.197904    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz582\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-kube-api-access-jz582\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.198462 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.198441    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-registry-certificates\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.199150 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.199125    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-trusted-ca\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.199843 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.199824    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-image-registry-private-configuration\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.200097 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.200078    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-installation-pull-secrets\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.205136 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.205114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"
Sep 07 22:33:08.205677 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.205653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-bound-sa-token\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.206070 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.206052    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz582\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-kube-api-access-jz582\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.298808 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.298774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cd7b3adc-224b-4509-a86c-a892d692883d-snapshots\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.298808 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.298807    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7b3adc-224b-4509-a86c-a892d692883d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.298808 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.298825    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd7b3adc-224b-4509-a86c-a892d692883d-serving-cert\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.299089 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.298841    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7b3adc-224b-4509-a86c-a892d692883d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.299089 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.299019    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jm7ls\" (UniqueName: \"kubernetes.io/projected/cd7b3adc-224b-4509-a86c-a892d692883d-kube-api-access-jm7ls\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.299191 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.299100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd7b3adc-224b-4509-a86c-a892d692883d-tmp\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.299581 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.299558    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cd7b3adc-224b-4509-a86c-a892d692883d-tmp\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.299699 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.299604    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/cd7b3adc-224b-4509-a86c-a892d692883d-snapshots\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.299699 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.299608    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7b3adc-224b-4509-a86c-a892d692883d-service-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.300374 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.300350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cd7b3adc-224b-4509-a86c-a892d692883d-trusted-ca-bundle\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.301464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.301444    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/cd7b3adc-224b-4509-a86c-a892d692883d-serving-cert\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.307293 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.307260    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jm7ls\" (UniqueName: \"kubernetes.io/projected/cd7b3adc-224b-4509-a86c-a892d692883d-kube-api-access-jm7ls\") pod \"insights-operator-76bcd6ccf5-mkfjw\" (UID: \"cd7b3adc-224b-4509-a86c-a892d692883d\") " pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.398846 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.398808    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw"
Sep 07 22:33:08.471041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.470965    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78"]
Sep 07 22:33:08.474624 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:08.474600    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76d268f3_0e43_4b16_8386_cd3c4e9a6ee8.slice/crio-f3f7d3142d929d45be071ecb6a242dcbb7367f326834c14ed4aabbb5249ddc52 WatchSource:0}: Error finding container f3f7d3142d929d45be071ecb6a242dcbb7367f326834c14ed4aabbb5249ddc52: Status 404 returned error can't find the container with id f3f7d3142d929d45be071ecb6a242dcbb7367f326834c14ed4aabbb5249ddc52
Sep 07 22:33:08.662353 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.662318    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-76bcd6ccf5-mkfjw"]
Sep 07 22:33:08.666260 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:08.666230    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd7b3adc_224b_4509_a86c_a892d692883d.slice/crio-b3703cba6ff82ac0520b40faa188e005fdf5667e73a3580da9e9b3d115c450ab WatchSource:0}: Error finding container b3703cba6ff82ac0520b40faa188e005fdf5667e73a3580da9e9b3d115c450ab: Status 404 returned error can't find the container with id b3703cba6ff82ac0520b40faa188e005fdf5667e73a3580da9e9b3d115c450ab
Sep 07 22:33:08.702498 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:08.702463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:08.702648 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:08.702612    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:33:08.702648 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:08.702634    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf8f549f5-6r27h: secret "image-registry-tls" not found
Sep 07 22:33:08.702724 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:08.702686    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls podName:4e368b88-1ea7-45f5-87a1-f83802318365 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:09.702670813 +0000 UTC m=+56.168827786 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls") pod "image-registry-6cf8f549f5-6r27h" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365") : secret "image-registry-tls" not found
Sep 07 22:33:09.323224 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:09.323138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw" event={"ID":"cd7b3adc-224b-4509-a86c-a892d692883d","Type":"ContainerStarted","Data":"b3703cba6ff82ac0520b40faa188e005fdf5667e73a3580da9e9b3d115c450ab"}
Sep 07 22:33:09.324792 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:09.324765    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78" event={"ID":"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8","Type":"ContainerStarted","Data":"f3f7d3142d929d45be071ecb6a242dcbb7367f326834c14ed4aabbb5249ddc52"}
Sep 07 22:33:09.711542 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:09.711434    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:09.711706 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:09.711583    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:33:09.711706 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:09.711599    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf8f549f5-6r27h: secret "image-registry-tls" not found
Sep 07 22:33:09.711706 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:09.711666    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls podName:4e368b88-1ea7-45f5-87a1-f83802318365 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:11.711646812 +0000 UTC m=+58.177803791 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls") pod "image-registry-6cf8f549f5-6r27h" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365") : secret "image-registry-tls" not found
Sep 07 22:33:11.728255 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:11.728176    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:11.728693 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:11.728313    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:33:11.728693 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:11.728325    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf8f549f5-6r27h: secret "image-registry-tls" not found
Sep 07 22:33:11.728693 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:11.728378    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls podName:4e368b88-1ea7-45f5-87a1-f83802318365 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:15.728364142 +0000 UTC m=+62.194521114 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls") pod "image-registry-6cf8f549f5-6r27h" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365") : secret "image-registry-tls" not found
Sep 07 22:33:12.334872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.334832    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78" event={"ID":"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8","Type":"ContainerStarted","Data":"f25a8c95da20d983fe9558fec308f43ad523cee42b56e2b2bfe53de0e9e8320b"}
Sep 07 22:33:12.336979 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.336950    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw" event={"ID":"cd7b3adc-224b-4509-a86c-a892d692883d","Type":"ContainerStarted","Data":"32ee56e171f39602d50ebc7b7ef3daef77b2486fc198592960ac2ab7248256e3"}
Sep 07 22:33:12.354009 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.353944    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78" podStartSLOduration=2.428312586 podStartE2EDuration="5.353922558s" totalImagesPullingTime="2.925609972s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:12.351889417 +0000 UTC m=+58.818046409" watchObservedRunningTime="2026-09-07 22:33:12.353922558 +0000 UTC m=+58.820079555"
Sep 07 22:33:12.367147 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.367099    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw" podStartSLOduration=1.628307373 podStartE2EDuration="4.367085154s" totalImagesPullingTime="2.738777781s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:12.365952804 +0000 UTC m=+58.832109801" watchObservedRunningTime="2026-09-07 22:33:12.367085154 +0000 UTC m=+58.833242195"
Sep 07 22:33:12.570757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.570720    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxns"]
Sep 07 22:33:12.593403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.593312    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxns"]
Sep 07 22:33:12.593542 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.593429    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.595879 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.595852    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 07 22:33:12.595879 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.595852    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 07 22:33:12.596051 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.595931    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tb8g9\""
Sep 07 22:33:12.735302 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.735246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/032352dd-0f32-454b-8363-69b634f412bd-data-volume\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.735741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.735326    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/032352dd-0f32-454b-8363-69b634f412bd-crio-socket\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.735741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.735385    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bn766\" (UniqueName: \"kubernetes.io/projected/032352dd-0f32-454b-8363-69b634f412bd-kube-api-access-bn766\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.735741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.735462    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.735741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.735493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.836588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.836543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/032352dd-0f32-454b-8363-69b634f412bd-crio-socket\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.836588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.836587    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bn766\" (UniqueName: \"kubernetes.io/projected/032352dd-0f32-454b-8363-69b634f412bd-kube-api-access-bn766\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.836835 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.836625    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.836835 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.836650    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.836835 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.836697    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/032352dd-0f32-454b-8363-69b634f412bd-crio-socket\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.836835 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:12.836759    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:12.836835 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:12.836772    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:12.836835 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:12.836819    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:13.336804737 +0000 UTC m=+59.802961710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:12.836835 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:12.836833    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:13.33682674 +0000 UTC m=+59.802983712 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:12.837094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.836848    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/032352dd-0f32-454b-8363-69b634f412bd-data-volume\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.837147 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.837132    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/032352dd-0f32-454b-8363-69b634f412bd-data-volume\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:12.844519 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:12.844445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bn766\" (UniqueName: \"kubernetes.io/projected/032352dd-0f32-454b-8363-69b634f412bd-kube-api-access-bn766\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:13.252728 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:13.252655    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-h6c9s"
Sep 07 22:33:13.341377 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:13.341345    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:13.341377 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:13.341382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:13.341622 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:13.341460    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:13.341622 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:13.341471    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:13.341622 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:13.341510    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:14.341495968 +0000 UTC m=+60.807652942 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:13.341622 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:13.341527    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:14.341515203 +0000 UTC m=+60.807672177 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:14.326739 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.326711    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-62ddd_6075fe40-91f5-4f32-ada2-e3a3ed7c91fc/dns-node-resolver/0.log"
Sep 07 22:33:14.350150 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.350117    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:14.350318 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.350153    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:14.350318 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:14.350243    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:14.350440 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:14.350320    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:16.350304503 +0000 UTC m=+62.816461480 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:14.350440 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:14.350333    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:14.350440 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:14.350390    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:16.350372846 +0000 UTC m=+62.816529835 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:14.420247 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.420216    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"]
Sep 07 22:33:14.422593 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.422578    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"
Sep 07 22:33:14.424990 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.424965    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 07 22:33:14.425149 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.425010    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-tk67g\""
Sep 07 22:33:14.425799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.425782    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 07 22:33:14.432043 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.432022    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"]
Sep 07 22:33:14.552141 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.552106    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj52w\" (UniqueName: \"kubernetes.io/projected/c92b7d60-6d95-472e-b96a-e29dea11ecf5-kube-api-access-dj52w\") pod \"migrator-fdbb8fdff-gvmg8\" (UID: \"c92b7d60-6d95-472e-b96a-e29dea11ecf5\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"
Sep 07 22:33:14.652765 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.652674    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj52w\" (UniqueName: \"kubernetes.io/projected/c92b7d60-6d95-472e-b96a-e29dea11ecf5-kube-api-access-dj52w\") pod \"migrator-fdbb8fdff-gvmg8\" (UID: \"c92b7d60-6d95-472e-b96a-e29dea11ecf5\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"
Sep 07 22:33:14.660048 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.660019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj52w\" (UniqueName: \"kubernetes.io/projected/c92b7d60-6d95-472e-b96a-e29dea11ecf5-kube-api-access-dj52w\") pod \"migrator-fdbb8fdff-gvmg8\" (UID: \"c92b7d60-6d95-472e-b96a-e29dea11ecf5\") " pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"
Sep 07 22:33:14.730799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.730768    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"
Sep 07 22:33:14.994669 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:14.994591    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8"]
Sep 07 22:33:14.997849 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:14.997824    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc92b7d60_6d95_472e_b96a_e29dea11ecf5.slice/crio-817ea6e7ae22c08f79abe0e011d5de9b121890fc76ea709e5c8fdee34422b869 WatchSource:0}: Error finding container 817ea6e7ae22c08f79abe0e011d5de9b121890fc76ea709e5c8fdee34422b869: Status 404 returned error can't find the container with id 817ea6e7ae22c08f79abe0e011d5de9b121890fc76ea709e5c8fdee34422b869
Sep 07 22:33:15.346110 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:15.346077    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8" event={"ID":"c92b7d60-6d95-472e-b96a-e29dea11ecf5","Type":"ContainerStarted","Data":"817ea6e7ae22c08f79abe0e011d5de9b121890fc76ea709e5c8fdee34422b869"}
Sep 07 22:33:15.527815 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:15.527786    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fljbl_434fae71-2d34-4e71-b384-83aa51cf0a28/node-ca/0.log"
Sep 07 22:33:15.761161 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:15.761077    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:15.761321 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:15.761236    2582 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 07 22:33:15.761321 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:15.761256    2582 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6cf8f549f5-6r27h: secret "image-registry-tls" not found
Sep 07 22:33:15.761396 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:15.761332    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls podName:4e368b88-1ea7-45f5-87a1-f83802318365 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:23.761315134 +0000 UTC m=+70.227472108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls") pod "image-registry-6cf8f549f5-6r27h" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365") : secret "image-registry-tls" not found
Sep 07 22:33:16.350980 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:16.350944    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8" event={"ID":"c92b7d60-6d95-472e-b96a-e29dea11ecf5","Type":"ContainerStarted","Data":"5264cb840d3d88203b89a5758d9f1d7fbe17b13738d774d41ae57dcff1f0ccca"}
Sep 07 22:33:16.351395 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:16.351004    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8" event={"ID":"c92b7d60-6d95-472e-b96a-e29dea11ecf5","Type":"ContainerStarted","Data":"e5fa4c44405a594c1340d36c3afd14b31d6c6ee06d4d39c85eccf67358dcfe66"}
Sep 07 22:33:16.365912 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:16.365860    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-fdbb8fdff-gvmg8" podStartSLOduration=1.356303458 podStartE2EDuration="2.365843096s" totalImagesPullingTime="1.009539638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:16.365276273 +0000 UTC m=+62.831433268" watchObservedRunningTime="2026-09-07 22:33:16.365843096 +0000 UTC m=+62.832000090"
Sep 07 22:33:16.366523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:16.366503    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:16.366583 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:16.366563    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:16.366648 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:16.366637    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:16.366684 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:16.366646    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:16.366684 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:16.366682    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:20.366671742 +0000 UTC m=+66.832828715 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:16.366750 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:16.366695    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:20.366689121 +0000 UTC m=+66.832846094 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:16.527267 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:16.527227    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nr7g8_48ba9096-4eaf-4076-bba7-4e6cecd4d23c/serve-healthcheck-canary/0.log"
Sep 07 22:33:19.694303 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.694251    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:33:19.694830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.694342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:33:19.694830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.694453    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:33:19.694830 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:19.694483    2582 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 07 22:33:19.694830 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:19.694553    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls podName:12799779-c22a-4ead-829f-e75c7eff4644 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:51.694539442 +0000 UTC m=+98.160696415 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls") pod "dns-default-tthjd" (UID: "12799779-c22a-4ead-829f-e75c7eff4644") : secret "dns-default-metrics-tls" not found
Sep 07 22:33:19.696627 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.696609    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 07 22:33:19.696796 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.696708    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 07 22:33:19.704746 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:19.704719    2582 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 07 22:33:19.704843 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:19.704771    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs podName:cc6804dc-8637-4596-8769-222cc116238e nodeName:}" failed. No retries permitted until 2026-09-07 22:34:23.704757358 +0000 UTC m=+130.170914353 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs") pod "network-metrics-daemon-4sqns" (UID: "cc6804dc-8637-4596-8769-222cc116238e") : secret "metrics-daemon-secret" not found
Sep 07 22:33:19.706776 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.706759    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/a679bbf7-133d-425c-959d-8eb901ecdcf4-original-pull-secret\") pod \"global-pull-secret-syncer-22swc\" (UID: \"a679bbf7-133d-425c-959d-8eb901ecdcf4\") " pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:33:19.794927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.794884    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:33:19.797294 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.797259    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 07 22:33:19.807110 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.807087    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 07 22:33:19.818450 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.818419    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbhzd\" (UniqueName: \"kubernetes.io/projected/22f7133d-4d2a-4e79-acfc-44402fd0b947-kube-api-access-lbhzd\") pod \"network-check-target-vrxqj\" (UID: \"22f7133d-4d2a-4e79-acfc-44402fd0b947\") " pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:33:19.897647 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.897593    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-22swc"
Sep 07 22:33:19.912440 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.912414    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-7qh6q\""
Sep 07 22:33:19.921318 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:19.921261    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:33:20.164694 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:20.164664    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-22swc"]
Sep 07 22:33:20.167408 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:20.167380    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda679bbf7_133d_425c_959d_8eb901ecdcf4.slice/crio-a49f075cdde7e64b055ab32d27218f98a19b12220f2a56eddf5187ee31ed651f WatchSource:0}: Error finding container a49f075cdde7e64b055ab32d27218f98a19b12220f2a56eddf5187ee31ed651f: Status 404 returned error can't find the container with id a49f075cdde7e64b055ab32d27218f98a19b12220f2a56eddf5187ee31ed651f
Sep 07 22:33:20.200555 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:20.199628    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-vrxqj"]
Sep 07 22:33:20.363045 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:20.363006    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vrxqj" event={"ID":"22f7133d-4d2a-4e79-acfc-44402fd0b947","Type":"ContainerStarted","Data":"9bd752c672c31d7368597bfa711db5c9ab2c934133bdac2043bf823b69f408e6"}
Sep 07 22:33:20.364187 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:20.364153    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-22swc" event={"ID":"a679bbf7-133d-425c-959d-8eb901ecdcf4","Type":"ContainerStarted","Data":"a49f075cdde7e64b055ab32d27218f98a19b12220f2a56eddf5187ee31ed651f"}
Sep 07 22:33:20.402719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:20.402685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:20.402832 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:20.402775    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:20.402883 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:20.402854    2582 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 07 22:33:20.402920 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:20.402913    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:28.402898867 +0000 UTC m=+74.869055840 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : configmap "kube-rbac-proxy" not found
Sep 07 22:33:20.402962 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:20.402856    2582 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:20.403005 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:20.402994    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls podName:032352dd-0f32-454b-8363-69b634f412bd nodeName:}" failed. No retries permitted until 2026-09-07 22:33:28.402979985 +0000 UTC m=+74.869136972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls") pod "insights-runtime-extractor-8mxns" (UID: "032352dd-0f32-454b-8363-69b634f412bd") : secret "insights-runtime-extractor-tls" not found
Sep 07 22:33:23.832779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:23.832739    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:23.835439 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:23.835411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"image-registry-6cf8f549f5-6r27h\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") " pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:23.909397 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:23.909355    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-hvdml\""
Sep 07 22:33:23.917211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:23.917169    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:24.879097 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:24.878963    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6cf8f549f5-6r27h"]
Sep 07 22:33:24.883853 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:24.883828    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e368b88_1ea7_45f5_87a1_f83802318365.slice/crio-6cf9c60a6ff4d21f2fd2c8f32ab6dfafe4d60d7c2dd7e2d5ca842a31751e6ee8 WatchSource:0}: Error finding container 6cf9c60a6ff4d21f2fd2c8f32ab6dfafe4d60d7c2dd7e2d5ca842a31751e6ee8: Status 404 returned error can't find the container with id 6cf9c60a6ff4d21f2fd2c8f32ab6dfafe4d60d7c2dd7e2d5ca842a31751e6ee8
Sep 07 22:33:25.381609 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.381567    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-vrxqj" event={"ID":"22f7133d-4d2a-4e79-acfc-44402fd0b947","Type":"ContainerStarted","Data":"36092fc176c56eb27fe0498041ec0c3020c3ed94185e44bf843e370b1bbf4a6f"}
Sep 07 22:33:25.381796 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.381679    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:33:25.383205 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.383169    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-22swc" event={"ID":"a679bbf7-133d-425c-959d-8eb901ecdcf4","Type":"ContainerStarted","Data":"a9ced3f8cae2e08cf85e2561e2e4a17912f238ddf6e570e7967b13ce0466b608"}
Sep 07 22:33:25.384724 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.384703    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h" event={"ID":"4e368b88-1ea7-45f5-87a1-f83802318365","Type":"ContainerStarted","Data":"2f75d8a9aabe26ffb1e4f087d8a7864807d3015f1f9dce3465909d99c2b9cbb0"}
Sep 07 22:33:25.384818 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.384736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h" event={"ID":"4e368b88-1ea7-45f5-87a1-f83802318365","Type":"ContainerStarted","Data":"6cf9c60a6ff4d21f2fd2c8f32ab6dfafe4d60d7c2dd7e2d5ca842a31751e6ee8"}
Sep 07 22:33:25.384818 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.384769    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:25.395488 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.395441    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-vrxqj" podStartSLOduration=66.984314733 podStartE2EDuration="1m11.395424645s" totalImagesPullingTime="4.411109912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:25.394412538 +0000 UTC m=+71.860569537" watchObservedRunningTime="2026-09-07 22:33:25.395424645 +0000 UTC m=+71.861581641"
Sep 07 22:33:25.410638 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.410588    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h" podStartSLOduration=18.410571414 podStartE2EDuration="18.410571414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:25.409673999 +0000 UTC m=+71.875830993" watchObservedRunningTime="2026-09-07 22:33:25.410571414 +0000 UTC m=+71.876728408"
Sep 07 22:33:25.422749 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:25.422700    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-22swc" podStartSLOduration=66.970279336 podStartE2EDuration="1m11.422684679s" totalImagesPullingTime="4.452405343s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:25.422247864 +0000 UTC m=+71.888404860" watchObservedRunningTime="2026-09-07 22:33:25.422684679 +0000 UTC m=+71.888841675"
Sep 07 22:33:28.469474 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:28.469433    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:28.469959 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:28.469497    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:28.470112 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:28.470087    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/032352dd-0f32-454b-8363-69b634f412bd-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:28.471700 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:28.471683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/032352dd-0f32-454b-8363-69b634f412bd-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-8mxns\" (UID: \"032352dd-0f32-454b-8363-69b634f412bd\") " pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:28.504880 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:28.504853    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-tb8g9\""
Sep 07 22:33:28.513633 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:28.513615    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-8mxns"
Sep 07 22:33:28.785537 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:28.785451    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-8mxns"]
Sep 07 22:33:28.788321 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:28.788269    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032352dd_0f32_454b_8363_69b634f412bd.slice/crio-860af6f16d0a8ec059aadf8bec4999ac3539b43f899d0c03854f1a0af3173e8c WatchSource:0}: Error finding container 860af6f16d0a8ec059aadf8bec4999ac3539b43f899d0c03854f1a0af3173e8c: Status 404 returned error can't find the container with id 860af6f16d0a8ec059aadf8bec4999ac3539b43f899d0c03854f1a0af3173e8c
Sep 07 22:33:29.400239 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:29.400196    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxns" event={"ID":"032352dd-0f32-454b-8363-69b634f412bd","Type":"ContainerStarted","Data":"ca03b8a387e65a51c5de77b5cbce8d0d820a933cc96890a5e06bf6c1581f8c5a"}
Sep 07 22:33:29.400449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:29.400271    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxns" event={"ID":"032352dd-0f32-454b-8363-69b634f412bd","Type":"ContainerStarted","Data":"860af6f16d0a8ec059aadf8bec4999ac3539b43f899d0c03854f1a0af3173e8c"}
Sep 07 22:33:30.405596 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:30.405547    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxns" event={"ID":"032352dd-0f32-454b-8363-69b634f412bd","Type":"ContainerStarted","Data":"6a84f9fe2f455f6b0e23abc4b6536fea2bb1daca868e60e7ac7d30828081176d"}
Sep 07 22:33:31.411343 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:31.411305    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-8mxns" event={"ID":"032352dd-0f32-454b-8363-69b634f412bd","Type":"ContainerStarted","Data":"e2c0f08ba350489e5bfd1093d1e328f95b8e9238e9bf6dff18552f919791294c"}
Sep 07 22:33:31.428366 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:31.428318    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-8mxns" podStartSLOduration=17.274745733 podStartE2EDuration="19.42829998s" totalImagesPullingTime="2.153554247s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 22:33:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:31.428185053 +0000 UTC m=+77.894342047" watchObservedRunningTime="2026-09-07 22:33:31.42829998 +0000 UTC m=+77.894456965"
Sep 07 22:33:37.319913 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.319789    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cf8f549f5-6r27h"]
Sep 07 22:33:37.345343 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.345312    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-67fc699b4f-d9zgs"]
Sep 07 22:33:37.348424 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.348405    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.351492 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.351467    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"]
Sep 07 22:33:37.354165 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.354148    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.358336 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.358317    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 07 22:33:37.358443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.358316    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 07 22:33:37.359401 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.359382    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 07 22:33:37.363477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.363458    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 07 22:33:37.364865 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.364850    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 07 22:33:37.365075 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.365057    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 07 22:33:37.365224 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.365207    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 07 22:33:37.369839 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.369815    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67fc699b4f-d9zgs"]
Sep 07 22:33:37.377964 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.377940    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"]
Sep 07 22:33:37.435250 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-hub\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.435250 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435253    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/659fb193-89e8-49e2-9807-c5d133251ce8-registry-certificates\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435470 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-ca\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.435470 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435350    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpqf9\" (UniqueName: \"kubernetes.io/projected/e3ed75d2-3c97-474b-960c-9918f757710f-kube-api-access-mpqf9\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.435470 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435424    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/659fb193-89e8-49e2-9807-c5d133251ce8-installation-pull-secrets\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435470 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/659fb193-89e8-49e2-9807-c5d133251ce8-image-registry-private-configuration\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435591 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-bound-sa-token\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435591 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435509    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ed75d2-3c97-474b-960c-9918f757710f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.435591 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/659fb193-89e8-49e2-9807-c5d133251ce8-ca-trust-extracted\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435591 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435550    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/659fb193-89e8-49e2-9807-c5d133251ce8-trusted-ca\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435711 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435593    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-registry-tls\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435711 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.435711 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435641    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lkftk\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-kube-api-access-lkftk\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.435711 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.435684    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.536304 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536249    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-hub\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.536503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536315    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/659fb193-89e8-49e2-9807-c5d133251ce8-registry-certificates\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.536503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-ca\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.536503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536455    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpqf9\" (UniqueName: \"kubernetes.io/projected/e3ed75d2-3c97-474b-960c-9918f757710f-kube-api-access-mpqf9\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.536503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536501    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/659fb193-89e8-49e2-9807-c5d133251ce8-installation-pull-secrets\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.536719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536547    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/659fb193-89e8-49e2-9807-c5d133251ce8-image-registry-private-configuration\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.536719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536574    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-bound-sa-token\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.536719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536600    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ed75d2-3c97-474b-960c-9918f757710f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.536719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536633    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/659fb193-89e8-49e2-9807-c5d133251ce8-ca-trust-extracted\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.536719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536665    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/659fb193-89e8-49e2-9807-c5d133251ce8-trusted-ca\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.536719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536691    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-registry-tls\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.536719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536717    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.537062 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536749    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lkftk\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-kube-api-access-lkftk\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.537062 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.536792    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.537268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.537244    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/659fb193-89e8-49e2-9807-c5d133251ce8-registry-certificates\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.538468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.537820    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/659fb193-89e8-49e2-9807-c5d133251ce8-trusted-ca\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.538468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.537945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3ed75d2-3c97-474b-960c-9918f757710f-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.538468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.538131    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/659fb193-89e8-49e2-9807-c5d133251ce8-ca-trust-extracted\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.539437 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.539378    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-ca\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.539782 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.539759    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/659fb193-89e8-49e2-9807-c5d133251ce8-image-registry-private-configuration\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.539965 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.539944    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-hub\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.540191 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.540169    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.540454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.540435    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/659fb193-89e8-49e2-9807-c5d133251ce8-installation-pull-secrets\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.540454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.540448    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-registry-tls\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.540672 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.540644    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e3ed75d2-3c97-474b-960c-9918f757710f-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.544917 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.544890    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-bound-sa-token\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.546033 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.546013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lkftk\" (UniqueName: \"kubernetes.io/projected/659fb193-89e8-49e2-9807-c5d133251ce8-kube-api-access-lkftk\") pod \"image-registry-67fc699b4f-d9zgs\" (UID: \"659fb193-89e8-49e2-9807-c5d133251ce8\") " pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.550797 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.550772    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpqf9\" (UniqueName: \"kubernetes.io/projected/e3ed75d2-3c97-474b-960c-9918f757710f-kube-api-access-mpqf9\") pod \"cluster-proxy-proxy-agent-5845f8bd6c-txtgz\" (UID: \"e3ed75d2-3c97-474b-960c-9918f757710f\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.657349 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.657255    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:37.674782 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.674750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"
Sep 07 22:33:37.937788 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.937711    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-67fc699b4f-d9zgs"]
Sep 07 22:33:37.940920 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:37.940882    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod659fb193_89e8_49e2_9807_c5d133251ce8.slice/crio-e827c3719fcfd4de789d7ae6522f2db458cea2d8c8a18485f3abb1f4658203e8 WatchSource:0}: Error finding container e827c3719fcfd4de789d7ae6522f2db458cea2d8c8a18485f3abb1f4658203e8: Status 404 returned error can't find the container with id e827c3719fcfd4de789d7ae6522f2db458cea2d8c8a18485f3abb1f4658203e8
Sep 07 22:33:37.959043 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:37.959015    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz"]
Sep 07 22:33:37.960982 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:37.960957    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode3ed75d2_3c97_474b_960c_9918f757710f.slice/crio-037e4b97dc707c7b79284a57339c83b0eb522bd8fc0dd0e48668be9f805e6f0d WatchSource:0}: Error finding container 037e4b97dc707c7b79284a57339c83b0eb522bd8fc0dd0e48668be9f805e6f0d: Status 404 returned error can't find the container with id 037e4b97dc707c7b79284a57339c83b0eb522bd8fc0dd0e48668be9f805e6f0d
Sep 07 22:33:38.435978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:38.435942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz" event={"ID":"e3ed75d2-3c97-474b-960c-9918f757710f","Type":"ContainerStarted","Data":"037e4b97dc707c7b79284a57339c83b0eb522bd8fc0dd0e48668be9f805e6f0d"}
Sep 07 22:33:38.437380 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:38.437359    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs" event={"ID":"659fb193-89e8-49e2-9807-c5d133251ce8","Type":"ContainerStarted","Data":"c5561705a3e1c371267ba082a8fdd714da164a5aaf1e2e0cc688340cd42a39d4"}
Sep 07 22:33:38.437457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:38.437400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs" event={"ID":"659fb193-89e8-49e2-9807-c5d133251ce8","Type":"ContainerStarted","Data":"e827c3719fcfd4de789d7ae6522f2db458cea2d8c8a18485f3abb1f4658203e8"}
Sep 07 22:33:38.437495 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:38.437476    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:33:38.455788 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:38.455735    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs" podStartSLOduration=1.455714896 podStartE2EDuration="1.455714896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:38.454357975 +0000 UTC m=+84.920514975" watchObservedRunningTime="2026-09-07 22:33:38.455714896 +0000 UTC m=+84.921871890"
Sep 07 22:33:42.454215 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:42.454177    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz" event={"ID":"e3ed75d2-3c97-474b-960c-9918f757710f","Type":"ContainerStarted","Data":"590fad113489b394d49c7d1e76c74027f2f1603101e6d7c30a7b728d9d5fe0c9"}
Sep 07 22:33:42.454576 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:42.454227    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz" event={"ID":"e3ed75d2-3c97-474b-960c-9918f757710f","Type":"ContainerStarted","Data":"81d363310a43880050224b7c210e3319a7ce649b30b1ed7dc06a2167be792e4c"}
Sep 07 22:33:42.454576 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:42.454241    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz" event={"ID":"e3ed75d2-3c97-474b-960c-9918f757710f","Type":"ContainerStarted","Data":"aa56f58c812a7a9a5c1c179a6402e21504a7cb99f8c84399950259392b18032d"}
Sep 07 22:33:42.472796 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:42.472747    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5845f8bd6c-txtgz" podStartSLOduration=1.500009377 podStartE2EDuration="5.472731889s" totalImagesPullingTime="3.972722512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:33:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:42.472394156 +0000 UTC m=+88.938551156" watchObservedRunningTime="2026-09-07 22:33:42.472731889 +0000 UTC m=+88.938888885"
Sep 07 22:33:47.324863 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:47.324821    2582 patch_prober.go:28] interesting pod/image-registry-6cf8f549f5-6r27h container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 07 22:33:47.325236 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:47.324888    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h" podUID="4e368b88-1ea7-45f5-87a1-f83802318365" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:33:51.752019 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:51.751946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:33:51.754391 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:51.754368    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12799779-c22a-4ead-829f-e75c7eff4644-metrics-tls\") pod \"dns-default-tthjd\" (UID: \"12799779-c22a-4ead-829f-e75c7eff4644\") " pod="openshift-dns/dns-default-tthjd"
Sep 07 22:33:51.808504 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:51.808469    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-zd2nk\""
Sep 07 22:33:51.816530 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:51.816491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-tthjd"
Sep 07 22:33:52.088561 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.088536    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-tthjd"]
Sep 07 22:33:52.089452 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:52.089422    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12799779_c22a_4ead_829f_e75c7eff4644.slice/crio-e6fb907ef7cbe327a1fa15fc86a04685d7a9e37617121e7a3aa89b00206c860a WatchSource:0}: Error finding container e6fb907ef7cbe327a1fa15fc86a04685d7a9e37617121e7a3aa89b00206c860a: Status 404 returned error can't find the container with id e6fb907ef7cbe327a1fa15fc86a04685d7a9e37617121e7a3aa89b00206c860a
Sep 07 22:33:52.488529 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.488443    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tthjd" event={"ID":"12799779-c22a-4ead-829f-e75c7eff4644","Type":"ContainerStarted","Data":"e6fb907ef7cbe327a1fa15fc86a04685d7a9e37617121e7a3aa89b00206c860a"}
Sep 07 22:33:52.955883 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.955846    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-mlgml"]
Sep 07 22:33:52.961630 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.961605    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:52.964251 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.964225    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-dzc2k\""
Sep 07 22:33:52.964402 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.964310    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 07 22:33:52.965175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.965150    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 07 22:33:52.965624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.965353    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 07 22:33:52.965624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.965350    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 07 22:33:52.965624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.965479    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 07 22:33:52.965861 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:52.965692    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 07 22:33:53.063619 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063581    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/557864a2-d5cb-47bf-bfad-8538d655e028-metrics-client-ca\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.063803 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-sys\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.063803 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063683    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.063803 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063708    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-root\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.063803 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063727    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/557864a2-d5cb-47bf-bfad-8538d655e028-kube-api-access-kpdkm\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.064015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063819    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-tls\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.064015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063855    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-wtmp\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.064015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063913    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.064015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.063946    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-textfile\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164517 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164485    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-root\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164526    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/557864a2-d5cb-47bf-bfad-8538d655e028-kube-api-access-kpdkm\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164576    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-tls\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164596    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-wtmp\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164597    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-root\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164635    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164663    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-textfile\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164706    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/557864a2-d5cb-47bf-bfad-8538d655e028-metrics-client-ca\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164981 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:53.164718    2582 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 07 22:33:53.164981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-wtmp\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164981 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:33:53.164778    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-tls podName:557864a2-d5cb-47bf-bfad-8538d655e028 nodeName:}" failed. No retries permitted until 2026-09-07 22:33:53.664759507 +0000 UTC m=+100.130916481 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-tls") pod "node-exporter-mlgml" (UID: "557864a2-d5cb-47bf-bfad-8538d655e028") : secret "node-exporter-tls" not found
Sep 07 22:33:53.164981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164872    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-sys\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.164981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.164905    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.165279 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.165019    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/557864a2-d5cb-47bf-bfad-8538d655e028-sys\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.165368 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.165325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-accelerators-collector-config\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.165423 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.165398    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/557864a2-d5cb-47bf-bfad-8538d655e028-metrics-client-ca\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.165634 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.165612    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-textfile\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.167541 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.167520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.172719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.172696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpdkm\" (UniqueName: \"kubernetes.io/projected/557864a2-d5cb-47bf-bfad-8538d655e028-kube-api-access-kpdkm\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.669584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.669530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-tls\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.671881 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.671854    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/557864a2-d5cb-47bf-bfad-8538d655e028-node-exporter-tls\") pod \"node-exporter-mlgml\" (UID: \"557864a2-d5cb-47bf-bfad-8538d655e028\") " pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:53.873057 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:53.873019    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-mlgml"
Sep 07 22:33:54.061513 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:33:54.061487    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557864a2_d5cb_47bf_bfad_8538d655e028.slice/crio-eb399d2e89832db00f19286c877cc02a151d05a0c70a62b7638e33c25e3834ff WatchSource:0}: Error finding container eb399d2e89832db00f19286c877cc02a151d05a0c70a62b7638e33c25e3834ff: Status 404 returned error can't find the container with id eb399d2e89832db00f19286c877cc02a151d05a0c70a62b7638e33c25e3834ff
Sep 07 22:33:54.496650 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:54.496599    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tthjd" event={"ID":"12799779-c22a-4ead-829f-e75c7eff4644","Type":"ContainerStarted","Data":"0b466f3ce2d3008d818119ae3873c4e25c79945b4c5eddea042a5edccad3659b"}
Sep 07 22:33:54.496845 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:54.496707    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-tthjd" event={"ID":"12799779-c22a-4ead-829f-e75c7eff4644","Type":"ContainerStarted","Data":"d878d991777e905d817fadb17fbe166feb7042d6c53167dec984f4dc718bad5f"}
Sep 07 22:33:54.496845 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:54.496737    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-tthjd"
Sep 07 22:33:54.497764 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:54.497741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgml" event={"ID":"557864a2-d5cb-47bf-bfad-8538d655e028","Type":"ContainerStarted","Data":"eb399d2e89832db00f19286c877cc02a151d05a0c70a62b7638e33c25e3834ff"}
Sep 07 22:33:54.513655 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:54.513590    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-tthjd" podStartSLOduration=65.544110542 podStartE2EDuration="1m7.513576091s" totalImagesPullingTime="1.969465549s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:54.512490824 +0000 UTC m=+100.978647839" watchObservedRunningTime="2026-09-07 22:33:54.513576091 +0000 UTC m=+100.979733086"
Sep 07 22:33:55.503489 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:55.503447    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgml" event={"ID":"557864a2-d5cb-47bf-bfad-8538d655e028","Type":"ContainerStarted","Data":"8198dc02a16def219e24aff2ac03f8190010202d2a5dc3747a07ab56ef73999b"}
Sep 07 22:33:56.390383 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:56.390354    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-vrxqj"
Sep 07 22:33:56.511531 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:56.511496    2582 generic.go:356] "Generic (PLEG): container finished" podID="557864a2-d5cb-47bf-bfad-8538d655e028" containerID="8198dc02a16def219e24aff2ac03f8190010202d2a5dc3747a07ab56ef73999b" exitCode=0
Sep 07 22:33:56.511953 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:56.511566    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgml" event={"ID":"557864a2-d5cb-47bf-bfad-8538d655e028","Type":"ContainerDied","Data":"8198dc02a16def219e24aff2ac03f8190010202d2a5dc3747a07ab56ef73999b"}
Sep 07 22:33:57.325027 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:57.325001    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:33:57.517133 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:57.517096    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgml" event={"ID":"557864a2-d5cb-47bf-bfad-8538d655e028","Type":"ContainerStarted","Data":"d5992452a81c8d8999866dd65dadc3cb631f36da18f06399249aa901fbad3755"}
Sep 07 22:33:57.517515 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:57.517160    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-mlgml" event={"ID":"557864a2-d5cb-47bf-bfad-8538d655e028","Type":"ContainerStarted","Data":"9732905fc620464a777b524ecd33b23500958ac94f7e709107b1bc1b7bc355e0"}
Sep 07 22:33:57.533813 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:57.533748    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-mlgml" podStartSLOduration=4.153484668 podStartE2EDuration="5.533731983s" totalImagesPullingTime="1.290364429s" totalInitContainerRuntime="89.882886ms" isStatefulPod=true podCreationTimestamp="2026-09-07 22:33:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:33:57.533589815 +0000 UTC m=+103.999746809" watchObservedRunningTime="2026-09-07 22:33:57.533731983 +0000 UTC m=+103.999888997"
Sep 07 22:33:59.446260 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:33:59.446229    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-67fc699b4f-d9zgs"
Sep 07 22:34:02.339754 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.339687    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h" podUID="4e368b88-1ea7-45f5-87a1-f83802318365" containerName="registry" containerID="cri-o://2f75d8a9aabe26ffb1e4f087d8a7864807d3015f1f9dce3465909d99c2b9cbb0" gracePeriod=30
Sep 07 22:34:02.537323 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.537269    2582 generic.go:356] "Generic (PLEG): container finished" podID="4e368b88-1ea7-45f5-87a1-f83802318365" containerID="2f75d8a9aabe26ffb1e4f087d8a7864807d3015f1f9dce3465909d99c2b9cbb0" exitCode=0
Sep 07 22:34:02.537494 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.537320    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h" event={"ID":"4e368b88-1ea7-45f5-87a1-f83802318365","Type":"ContainerDied","Data":"2f75d8a9aabe26ffb1e4f087d8a7864807d3015f1f9dce3465909d99c2b9cbb0"}
Sep 07 22:34:02.582906 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.582880    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:34:02.745417 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745321    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-bound-sa-token\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.745417 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745375    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-trusted-ca\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.745417 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745403    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-registry-certificates\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.745705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745425    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e368b88-1ea7-45f5-87a1-f83802318365-ca-trust-extracted\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.745705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745443    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-image-registry-private-configuration\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.745705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745472    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-installation-pull-secrets\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.745705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745512    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz582\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-kube-api-access-jz582\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.745705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745543    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") pod \"4e368b88-1ea7-45f5-87a1-f83802318365\" (UID: \"4e368b88-1ea7-45f5-87a1-f83802318365\") "
Sep 07 22:34:02.746006 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745919    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:34:02.746006 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.745976    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:34:02.748329 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.748299    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:34:02.748450 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.748302    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:34:02.748450 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.748378    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:34:02.748520 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.748480    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:34:02.748625 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.748606    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-kube-api-access-jz582" (OuterVolumeSpecName: "kube-api-access-jz582") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "kube-api-access-jz582". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:34:02.753727 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.753702    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4e368b88-1ea7-45f5-87a1-f83802318365-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "4e368b88-1ea7-45f5-87a1-f83802318365" (UID: "4e368b88-1ea7-45f5-87a1-f83802318365"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:34:02.846427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846382    2582 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-bound-sa-token\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:02.846427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846424    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-trusted-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:02.846427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846433    2582 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/4e368b88-1ea7-45f5-87a1-f83802318365-registry-certificates\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:02.846427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846443    2582 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/4e368b88-1ea7-45f5-87a1-f83802318365-ca-trust-extracted\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:02.846674 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846454    2582 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-image-registry-private-configuration\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:02.846674 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846463    2582 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/4e368b88-1ea7-45f5-87a1-f83802318365-installation-pull-secrets\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:02.846674 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846472    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz582\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-kube-api-access-jz582\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:02.846674 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:02.846481    2582 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/4e368b88-1ea7-45f5-87a1-f83802318365-registry-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:34:03.546146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:03.546107    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h" event={"ID":"4e368b88-1ea7-45f5-87a1-f83802318365","Type":"ContainerDied","Data":"6cf9c60a6ff4d21f2fd2c8f32ab6dfafe4d60d7c2dd7e2d5ca842a31751e6ee8"}
Sep 07 22:34:03.546146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:03.546148    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6cf8f549f5-6r27h"
Sep 07 22:34:03.546664 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:03.546199    2582 scope.go:117] "RemoveContainer" containerID="2f75d8a9aabe26ffb1e4f087d8a7864807d3015f1f9dce3465909d99c2b9cbb0"
Sep 07 22:34:03.565394 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:03.565368    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6cf8f549f5-6r27h"]
Sep 07 22:34:03.568956 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:03.568932    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6cf8f549f5-6r27h"]
Sep 07 22:34:04.088544 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:04.088504    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e368b88-1ea7-45f5-87a1-f83802318365" path="/var/lib/kubelet/pods/4e368b88-1ea7-45f5-87a1-f83802318365/volumes"
Sep 07 22:34:04.506045 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:04.505965    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-tthjd"
Sep 07 22:34:11.930075 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.930044    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-bb859669-rp4k4"]
Sep 07 22:34:11.930602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.930268    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e368b88-1ea7-45f5-87a1-f83802318365" containerName="registry"
Sep 07 22:34:11.930602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.930279    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e368b88-1ea7-45f5-87a1-f83802318365" containerName="registry"
Sep 07 22:34:11.930602 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.930337    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e368b88-1ea7-45f5-87a1-f83802318365" containerName="registry"
Sep 07 22:34:11.933152 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.933135    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:11.936445 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.936409    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 22:34:11.936566 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.936519    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 22:34:11.936613 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.936582    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 22:34:11.936613 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.936597    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 22:34:11.936680 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.936525    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 22:34:11.936741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.936726    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 22:34:11.937087 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.937073    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 22:34:11.937166 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.937132    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lhcs8\""
Sep 07 22:34:11.941826 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.941805    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 22:34:11.941961 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:11.941940    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb859669-rp4k4"]
Sep 07 22:34:12.122056 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.122025    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-trusted-ca-bundle\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.122232 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.122062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-config\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.122232 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.122079    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-oauth-config\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.122232 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.122096    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-oauth-serving-cert\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.122232 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.122165    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-serving-cert\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.122232 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.122218    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-service-ca\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.122427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.122236    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-km76w\" (UniqueName: \"kubernetes.io/projected/d278ccf1-786d-45fe-b078-8e05e6bad78e-kube-api-access-km76w\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.222771 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.222685    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-trusted-ca-bundle\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.222771 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.222730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-config\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.222771 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.222756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-oauth-config\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223030 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.222782    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-oauth-serving-cert\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223030 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.222808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-serving-cert\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223030 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.222838    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-service-ca\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223030 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.222995    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-km76w\" (UniqueName: \"kubernetes.io/projected/d278ccf1-786d-45fe-b078-8e05e6bad78e-kube-api-access-km76w\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223501 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.223475    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-config\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.223540    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-oauth-serving-cert\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223720 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.223696    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-trusted-ca-bundle\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.223826 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.223702    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-service-ca\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.225225 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.225203    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-serving-cert\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.225317 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.225241    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-oauth-config\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.230780 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.230755    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-km76w\" (UniqueName: \"kubernetes.io/projected/d278ccf1-786d-45fe-b078-8e05e6bad78e-kube-api-access-km76w\") pod \"console-bb859669-rp4k4\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") " pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.242578 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.242553    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:12.512046 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.511961    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-bb859669-rp4k4"]
Sep 07 22:34:12.515658 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:34:12.515633    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd278ccf1_786d_45fe_b078_8e05e6bad78e.slice/crio-57ad4812022271ce4462cd812887ccb4371c25b4a620a21827abee8cfe7404ea WatchSource:0}: Error finding container 57ad4812022271ce4462cd812887ccb4371c25b4a620a21827abee8cfe7404ea: Status 404 returned error can't find the container with id 57ad4812022271ce4462cd812887ccb4371c25b4a620a21827abee8cfe7404ea
Sep 07 22:34:12.583091 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:12.583048    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb859669-rp4k4" event={"ID":"d278ccf1-786d-45fe-b078-8e05e6bad78e","Type":"ContainerStarted","Data":"57ad4812022271ce4462cd812887ccb4371c25b4a620a21827abee8cfe7404ea"}
Sep 07 22:34:15.595660 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:15.595616    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb859669-rp4k4" event={"ID":"d278ccf1-786d-45fe-b078-8e05e6bad78e","Type":"ContainerStarted","Data":"2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10"}
Sep 07 22:34:15.614388 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:15.614327    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-bb859669-rp4k4" podStartSLOduration=1.962847462 podStartE2EDuration="4.614310377s" totalImagesPullingTime="2.651462915s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:34:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:15.613525413 +0000 UTC m=+122.079682410" watchObservedRunningTime="2026-09-07 22:34:15.614310377 +0000 UTC m=+122.080467371"
Sep 07 22:34:22.243142 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:22.243100    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:22.243584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:22.243172    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:22.248032 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:22.248006    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:22.623725 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:22.623686    2582 generic.go:356] "Generic (PLEG): container finished" podID="76d268f3-0e43-4b16-8386-cd3c4e9a6ee8" containerID="f25a8c95da20d983fe9558fec308f43ad523cee42b56e2b2bfe53de0e9e8320b" exitCode=0
Sep 07 22:34:22.623890 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:22.623758    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78" event={"ID":"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8","Type":"ContainerDied","Data":"f25a8c95da20d983fe9558fec308f43ad523cee42b56e2b2bfe53de0e9e8320b"}
Sep 07 22:34:22.624139 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:22.624122    2582 scope.go:117] "RemoveContainer" containerID="f25a8c95da20d983fe9558fec308f43ad523cee42b56e2b2bfe53de0e9e8320b"
Sep 07 22:34:22.627987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:22.627964    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:34:23.628929 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:23.628890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-859f94c4b-l9v78" event={"ID":"76d268f3-0e43-4b16-8386-cd3c4e9a6ee8","Type":"ContainerStarted","Data":"6d08fe0592a4b11d2c909382885a0af84a8e637f8f045357d027b23247c9e89f"}
Sep 07 22:34:23.716343 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:23.716190    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:34:23.718378 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:23.718355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cc6804dc-8637-4596-8769-222cc116238e-metrics-certs\") pod \"network-metrics-daemon-4sqns\" (UID: \"cc6804dc-8637-4596-8769-222cc116238e\") " pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:34:23.806666 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:23.806634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-lg5n9\""
Sep 07 22:34:23.814759 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:23.814737    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-4sqns"
Sep 07 22:34:24.080478 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:24.080438    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-4sqns"]
Sep 07 22:34:24.083278 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:34:24.083241    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc6804dc_8637_4596_8769_222cc116238e.slice/crio-55840717ab7776c9852ea4ff26b2fd5da955a5ddb3c6c9d351d2065d04f77585 WatchSource:0}: Error finding container 55840717ab7776c9852ea4ff26b2fd5da955a5ddb3c6c9d351d2065d04f77585: Status 404 returned error can't find the container with id 55840717ab7776c9852ea4ff26b2fd5da955a5ddb3c6c9d351d2065d04f77585
Sep 07 22:34:24.633826 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:24.633781    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sqns" event={"ID":"cc6804dc-8637-4596-8769-222cc116238e","Type":"ContainerStarted","Data":"55840717ab7776c9852ea4ff26b2fd5da955a5ddb3c6c9d351d2065d04f77585"}
Sep 07 22:34:25.641539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:25.641505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sqns" event={"ID":"cc6804dc-8637-4596-8769-222cc116238e","Type":"ContainerStarted","Data":"559af3901aa3435dff3d85f9e72d65f2ed9e55f2ff1850b09582c0cff928978f"}
Sep 07 22:34:25.641977 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:25.641562    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-4sqns" event={"ID":"cc6804dc-8637-4596-8769-222cc116238e","Type":"ContainerStarted","Data":"2f5d459a4c788598d2f55002ccac9635f2bf2e83c06976fbbaedb38c4f951f94"}
Sep 07 22:34:25.657928 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:25.657876    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-4sqns" podStartSLOduration=130.795980118 podStartE2EDuration="2m11.657858951s" totalImagesPullingTime="861.878833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:32:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:34:25.65653964 +0000 UTC m=+132.122696646" watchObservedRunningTime="2026-09-07 22:34:25.657858951 +0000 UTC m=+132.124015945"
Sep 07 22:34:37.681589 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:37.681552    2582 generic.go:356] "Generic (PLEG): container finished" podID="cd7b3adc-224b-4509-a86c-a892d692883d" containerID="32ee56e171f39602d50ebc7b7ef3daef77b2486fc198592960ac2ab7248256e3" exitCode=0
Sep 07 22:34:37.682015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:37.681597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw" event={"ID":"cd7b3adc-224b-4509-a86c-a892d692883d","Type":"ContainerDied","Data":"32ee56e171f39602d50ebc7b7ef3daef77b2486fc198592960ac2ab7248256e3"}
Sep 07 22:34:37.682015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:37.681933    2582 scope.go:117] "RemoveContainer" containerID="32ee56e171f39602d50ebc7b7ef3daef77b2486fc198592960ac2ab7248256e3"
Sep 07 22:34:38.687740 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:34:38.687704    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-76bcd6ccf5-mkfjw" event={"ID":"cd7b3adc-224b-4509-a86c-a892d692883d","Type":"ContainerStarted","Data":"6c1017f956e72bfa729ab5d80f3b6e4cd4477efdd147de9d3f3dfadf5803b2af"}
Sep 07 22:36:05.886016 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:05.885979    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb859669-rp4k4"]
Sep 07 22:36:25.941564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:25.941523    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"]
Sep 07 22:36:25.944681 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:25.944664    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:25.947193 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:25.947166    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 22:36:25.948178 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:25.948150    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rl8kx\""
Sep 07 22:36:25.948263 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:25.948173    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 22:36:25.954071 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:25.954049    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"]
Sep 07 22:36:26.032759 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.032714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.032932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.032774    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.032932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.032833    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk2qk\" (UniqueName: \"kubernetes.io/projected/9a2c47de-3e15-488f-a936-3536c1825dc3-kube-api-access-lk2qk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.133963 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.133928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.133963 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.133969    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.134189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.134103    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk2qk\" (UniqueName: \"kubernetes.io/projected/9a2c47de-3e15-488f-a936-3536c1825dc3-kube-api-access-lk2qk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.134416 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.134397    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.134458 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.134411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.142129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.142100    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk2qk\" (UniqueName: \"kubernetes.io/projected/9a2c47de-3e15-488f-a936-3536c1825dc3-kube-api-access-lk2qk\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.253192 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.253092    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:26.519068 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:26.519042    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"]
Sep 07 22:36:26.521583 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:36:26.521549    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a2c47de_3e15_488f_a936_3536c1825dc3.slice/crio-b9af3c88a9b1153c1efde40a0c5d96cfcd9e6e4c53f9d19049407e7233991726 WatchSource:0}: Error finding container b9af3c88a9b1153c1efde40a0c5d96cfcd9e6e4c53f9d19049407e7233991726: Status 404 returned error can't find the container with id b9af3c88a9b1153c1efde40a0c5d96cfcd9e6e4c53f9d19049407e7233991726
Sep 07 22:36:27.066035 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:27.065996    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw" event={"ID":"9a2c47de-3e15-488f-a936-3536c1825dc3","Type":"ContainerStarted","Data":"b9af3c88a9b1153c1efde40a0c5d96cfcd9e6e4c53f9d19049407e7233991726"}
Sep 07 22:36:30.905510 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:30.905462    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-bb859669-rp4k4" podUID="d278ccf1-786d-45fe-b078-8e05e6bad78e" containerName="console" containerID="cri-o://2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10" gracePeriod=15
Sep 07 22:36:31.989975 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:31.989945    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:36:32.083197 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083162    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-serving-cert\") pod \"d278ccf1-786d-45fe-b078-8e05e6bad78e\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") "
Sep 07 22:36:32.083545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083209    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-config\") pod \"d278ccf1-786d-45fe-b078-8e05e6bad78e\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") "
Sep 07 22:36:32.083545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083239    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-oauth-config\") pod \"d278ccf1-786d-45fe-b078-8e05e6bad78e\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") "
Sep 07 22:36:32.083545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-service-ca\") pod \"d278ccf1-786d-45fe-b078-8e05e6bad78e\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") "
Sep 07 22:36:32.083545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083426    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-oauth-serving-cert\") pod \"d278ccf1-786d-45fe-b078-8e05e6bad78e\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") "
Sep 07 22:36:32.083545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083461    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-trusted-ca-bundle\") pod \"d278ccf1-786d-45fe-b078-8e05e6bad78e\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") "
Sep 07 22:36:32.083545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083506    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-km76w\" (UniqueName: \"kubernetes.io/projected/d278ccf1-786d-45fe-b078-8e05e6bad78e-kube-api-access-km76w\") pod \"d278ccf1-786d-45fe-b078-8e05e6bad78e\" (UID: \"d278ccf1-786d-45fe-b078-8e05e6bad78e\") "
Sep 07 22:36:32.083892 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083668    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-config" (OuterVolumeSpecName: "console-config") pod "d278ccf1-786d-45fe-b078-8e05e6bad78e" (UID: "d278ccf1-786d-45fe-b078-8e05e6bad78e"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:36:32.083892 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083721    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-service-ca" (OuterVolumeSpecName: "service-ca") pod "d278ccf1-786d-45fe-b078-8e05e6bad78e" (UID: "d278ccf1-786d-45fe-b078-8e05e6bad78e"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:36:32.083892 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.083740    2582 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-config\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:32.084131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.084109    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "d278ccf1-786d-45fe-b078-8e05e6bad78e" (UID: "d278ccf1-786d-45fe-b078-8e05e6bad78e"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:36:32.084226 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.084152    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "d278ccf1-786d-45fe-b078-8e05e6bad78e" (UID: "d278ccf1-786d-45fe-b078-8e05e6bad78e"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:36:32.085820 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.085785    2582 generic.go:356] "Generic (PLEG): container finished" podID="d278ccf1-786d-45fe-b078-8e05e6bad78e" containerID="2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10" exitCode=0
Sep 07 22:36:32.085926 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.085867    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-bb859669-rp4k4"
Sep 07 22:36:32.086087 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.086062    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "d278ccf1-786d-45fe-b078-8e05e6bad78e" (UID: "d278ccf1-786d-45fe-b078-8e05e6bad78e"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:36:32.086209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.086087    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "d278ccf1-786d-45fe-b078-8e05e6bad78e" (UID: "d278ccf1-786d-45fe-b078-8e05e6bad78e"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:36:32.086209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.086182    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d278ccf1-786d-45fe-b078-8e05e6bad78e-kube-api-access-km76w" (OuterVolumeSpecName: "kube-api-access-km76w") pod "d278ccf1-786d-45fe-b078-8e05e6bad78e" (UID: "d278ccf1-786d-45fe-b078-8e05e6bad78e"). InnerVolumeSpecName "kube-api-access-km76w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:36:32.087925 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.087899    2582 generic.go:356] "Generic (PLEG): container finished" podID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerID="a49f0f424ce34182dd8cd71931e245097e31df5b1ed1a0c060d2c84f164f6001" exitCode=0
Sep 07 22:36:32.088902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.088877    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb859669-rp4k4" event={"ID":"d278ccf1-786d-45fe-b078-8e05e6bad78e","Type":"ContainerDied","Data":"2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10"}
Sep 07 22:36:32.088978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.088925    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-bb859669-rp4k4" event={"ID":"d278ccf1-786d-45fe-b078-8e05e6bad78e","Type":"ContainerDied","Data":"57ad4812022271ce4462cd812887ccb4371c25b4a620a21827abee8cfe7404ea"}
Sep 07 22:36:32.088978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.088940    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw" event={"ID":"9a2c47de-3e15-488f-a936-3536c1825dc3","Type":"ContainerDied","Data":"a49f0f424ce34182dd8cd71931e245097e31df5b1ed1a0c060d2c84f164f6001"}
Sep 07 22:36:32.089057 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.088964    2582 scope.go:117] "RemoveContainer" containerID="2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10"
Sep 07 22:36:32.097412 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.097392    2582 scope.go:117] "RemoveContainer" containerID="2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10"
Sep 07 22:36:32.097730 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:36:32.097706    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10\": container with ID starting with 2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10 not found: ID does not exist" containerID="2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10"
Sep 07 22:36:32.097799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.097739    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10"} err="failed to get container status \"2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10\": rpc error: code = NotFound desc = could not find container \"2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10\": container with ID starting with 2f5afa69dce382565d48331fbe25cb046e713f64a3e5a1036bc24f2b2940de10 not found: ID does not exist"
Sep 07 22:36:32.185129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.185032    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-km76w\" (UniqueName: \"kubernetes.io/projected/d278ccf1-786d-45fe-b078-8e05e6bad78e-kube-api-access-km76w\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:32.185129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.185063    2582 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-serving-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:32.185129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.185079    2582 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d278ccf1-786d-45fe-b078-8e05e6bad78e-console-oauth-config\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:32.185129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.185091    2582 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:32.185129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.185108    2582 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-oauth-serving-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:32.185129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.185120    2582 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d278ccf1-786d-45fe-b078-8e05e6bad78e-trusted-ca-bundle\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:32.401661 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.401630    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-bb859669-rp4k4"]
Sep 07 22:36:32.404795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:32.404771    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-bb859669-rp4k4"]
Sep 07 22:36:34.089505 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:34.089465    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d278ccf1-786d-45fe-b078-8e05e6bad78e" path="/var/lib/kubelet/pods/d278ccf1-786d-45fe-b078-8e05e6bad78e/volumes"
Sep 07 22:36:34.099073 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:34.099038    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw" event={"ID":"9a2c47de-3e15-488f-a936-3536c1825dc3","Type":"ContainerStarted","Data":"63f53d2356651f49d03a76350b42aeb2297b66f078cf2b2179396f848cdd081c"}
Sep 07 22:36:35.105195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:35.105153    2582 generic.go:356] "Generic (PLEG): container finished" podID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerID="63f53d2356651f49d03a76350b42aeb2297b66f078cf2b2179396f848cdd081c" exitCode=0
Sep 07 22:36:35.105689 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:35.105259    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw" event={"ID":"9a2c47de-3e15-488f-a936-3536c1825dc3","Type":"ContainerDied","Data":"63f53d2356651f49d03a76350b42aeb2297b66f078cf2b2179396f848cdd081c"}
Sep 07 22:36:42.134007 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:42.133971    2582 generic.go:356] "Generic (PLEG): container finished" podID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerID="f61069109988d1b2997ec2925fb8dce721835a272cba20b7d966126b624ab770" exitCode=0
Sep 07 22:36:42.134464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:42.134054    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw" event={"ID":"9a2c47de-3e15-488f-a936-3536c1825dc3","Type":"ContainerDied","Data":"f61069109988d1b2997ec2925fb8dce721835a272cba20b7d966126b624ab770"}
Sep 07 22:36:43.261081 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.261056    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:43.371810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.371771    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-bundle\") pod \"9a2c47de-3e15-488f-a936-3536c1825dc3\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") "
Sep 07 22:36:43.371810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.371811    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lk2qk\" (UniqueName: \"kubernetes.io/projected/9a2c47de-3e15-488f-a936-3536c1825dc3-kube-api-access-lk2qk\") pod \"9a2c47de-3e15-488f-a936-3536c1825dc3\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") "
Sep 07 22:36:43.372018 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.371842    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-util\") pod \"9a2c47de-3e15-488f-a936-3536c1825dc3\" (UID: \"9a2c47de-3e15-488f-a936-3536c1825dc3\") "
Sep 07 22:36:43.372254 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.372230    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-bundle" (OuterVolumeSpecName: "bundle") pod "9a2c47de-3e15-488f-a936-3536c1825dc3" (UID: "9a2c47de-3e15-488f-a936-3536c1825dc3"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:36:43.374012 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.373990    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a2c47de-3e15-488f-a936-3536c1825dc3-kube-api-access-lk2qk" (OuterVolumeSpecName: "kube-api-access-lk2qk") pod "9a2c47de-3e15-488f-a936-3536c1825dc3" (UID: "9a2c47de-3e15-488f-a936-3536c1825dc3"). InnerVolumeSpecName "kube-api-access-lk2qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:36:43.376630 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.376601    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-util" (OuterVolumeSpecName: "util") pod "9a2c47de-3e15-488f-a936-3536c1825dc3" (UID: "9a2c47de-3e15-488f-a936-3536c1825dc3"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:36:43.472628 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.472527    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-bundle\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:43.472628 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.472564    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk2qk\" (UniqueName: \"kubernetes.io/projected/9a2c47de-3e15-488f-a936-3536c1825dc3-kube-api-access-lk2qk\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:43.472628 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:43.472578    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/9a2c47de-3e15-488f-a936-3536c1825dc3-util\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:44.143005 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:44.142955    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw" event={"ID":"9a2c47de-3e15-488f-a936-3536c1825dc3","Type":"ContainerDied","Data":"b9af3c88a9b1153c1efde40a0c5d96cfcd9e6e4c53f9d19049407e7233991726"}
Sep 07 22:36:44.143005 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:44.142984    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fpzxqw"
Sep 07 22:36:44.143216 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:44.143015    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b9af3c88a9b1153c1efde40a0c5d96cfcd9e6e4c53f9d19049407e7233991726"
Sep 07 22:36:50.061405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061367    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"]
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061609    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerName="extract"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061622    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerName="extract"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061639    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerName="util"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061644    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerName="util"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061653    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d278ccf1-786d-45fe-b078-8e05e6bad78e" containerName="console"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061658    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="d278ccf1-786d-45fe-b078-8e05e6bad78e" containerName="console"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061665    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerName="pull"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061670    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerName="pull"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061709    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="d278ccf1-786d-45fe-b078-8e05e6bad78e" containerName="console"
Sep 07 22:36:50.061790 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.061716    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a2c47de-3e15-488f-a936-3536c1825dc3" containerName="extract"
Sep 07 22:36:50.101433 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.101391    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.104140 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.104110    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 07 22:36:50.105120 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.105095    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-xv2bm\""
Sep 07 22:36:50.105240 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.105095    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 07 22:36:50.106420 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.106402    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"]
Sep 07 22:36:50.219890 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.219859    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a332572-166b-4767-b357-7622d741c0c3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hj5jj\" (UID: \"9a332572-166b-4767-b357-7622d741c0c3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.220070 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.219896    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9j64c\" (UniqueName: \"kubernetes.io/projected/9a332572-166b-4767-b357-7622d741c0c3-kube-api-access-9j64c\") pod \"openshift-lws-operator-fd8ccff4c-hj5jj\" (UID: \"9a332572-166b-4767-b357-7622d741c0c3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.321120 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.321020    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a332572-166b-4767-b357-7622d741c0c3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hj5jj\" (UID: \"9a332572-166b-4767-b357-7622d741c0c3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.321120 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.321069    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9j64c\" (UniqueName: \"kubernetes.io/projected/9a332572-166b-4767-b357-7622d741c0c3-kube-api-access-9j64c\") pod \"openshift-lws-operator-fd8ccff4c-hj5jj\" (UID: \"9a332572-166b-4767-b357-7622d741c0c3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.321449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.321428    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/9a332572-166b-4767-b357-7622d741c0c3-tmp\") pod \"openshift-lws-operator-fd8ccff4c-hj5jj\" (UID: \"9a332572-166b-4767-b357-7622d741c0c3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.329138 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.329110    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9j64c\" (UniqueName: \"kubernetes.io/projected/9a332572-166b-4767-b357-7622d741c0c3-kube-api-access-9j64c\") pod \"openshift-lws-operator-fd8ccff4c-hj5jj\" (UID: \"9a332572-166b-4767-b357-7622d741c0c3\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.411440 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.411403    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"
Sep 07 22:36:50.698796 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:50.698769    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj"]
Sep 07 22:36:50.701277 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:36:50.701250    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9a332572_166b_4767_b357_7622d741c0c3.slice/crio-768ff315bbc69de0de4fd76705bbdd6b7746bae31a90b5c6d3af54e53a7ff00d WatchSource:0}: Error finding container 768ff315bbc69de0de4fd76705bbdd6b7746bae31a90b5c6d3af54e53a7ff00d: Status 404 returned error can't find the container with id 768ff315bbc69de0de4fd76705bbdd6b7746bae31a90b5c6d3af54e53a7ff00d
Sep 07 22:36:51.168478 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:51.168438    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj" event={"ID":"9a332572-166b-4767-b357-7622d741c0c3","Type":"ContainerStarted","Data":"768ff315bbc69de0de4fd76705bbdd6b7746bae31a90b5c6d3af54e53a7ff00d"}
Sep 07 22:36:53.177331 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:53.177295    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj" event={"ID":"9a332572-166b-4767-b357-7622d741c0c3","Type":"ContainerStarted","Data":"4773078dcca33f5ab10f2b0083f662b4dafc35aced51a35eb49119e8225449f5"}
Sep 07 22:36:53.195110 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:53.195050    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-hj5jj" podStartSLOduration=1.2173037660000001 podStartE2EDuration="3.195031765s" totalImagesPullingTime="1.977727999s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:36:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:36:53.194093168 +0000 UTC m=+279.660250157" watchObservedRunningTime="2026-09-07 22:36:53.195031765 +0000 UTC m=+279.661188760"
Sep 07 22:36:55.435524 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.435487    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"]
Sep 07 22:36:55.437642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.437624    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.439925 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.439906    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rl8kx\""
Sep 07 22:36:55.440153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.440139    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 22:36:55.441491 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.441472    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 22:36:55.446833 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.446805    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"]
Sep 07 22:36:55.561654 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.561617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.561654 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.561661    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.561891 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.561682    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hn8c2\" (UniqueName: \"kubernetes.io/projected/eedebdef-2a83-45de-8b29-a8cd04283e0e-kube-api-access-hn8c2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.662967 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.662925    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.663137 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.662976    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.663137 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.663005    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hn8c2\" (UniqueName: \"kubernetes.io/projected/eedebdef-2a83-45de-8b29-a8cd04283e0e-kube-api-access-hn8c2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.663347 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.663325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.663415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.663363    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.670149 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.670126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hn8c2\" (UniqueName: \"kubernetes.io/projected/eedebdef-2a83-45de-8b29-a8cd04283e0e-kube-api-access-hn8c2\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:55.747148 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:55.747056    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:56.017033 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:56.017004    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"]
Sep 07 22:36:56.019439 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:36:56.019408    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeedebdef_2a83_45de_8b29_a8cd04283e0e.slice/crio-86531b0face23d9dc1caae34ca0a25fa159d7e0edab945c2d9f44df4e9115ca0 WatchSource:0}: Error finding container 86531b0face23d9dc1caae34ca0a25fa159d7e0edab945c2d9f44df4e9115ca0: Status 404 returned error can't find the container with id 86531b0face23d9dc1caae34ca0a25fa159d7e0edab945c2d9f44df4e9115ca0
Sep 07 22:36:56.190204 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:56.190168    2582 generic.go:356] "Generic (PLEG): container finished" podID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerID="34b1eac5e236ef7faac339e55dc7b093cb1f174b6d7ee88e8c7187f8b5e599bd" exitCode=0
Sep 07 22:36:56.190401 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:56.190238    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc" event={"ID":"eedebdef-2a83-45de-8b29-a8cd04283e0e","Type":"ContainerDied","Data":"34b1eac5e236ef7faac339e55dc7b093cb1f174b6d7ee88e8c7187f8b5e599bd"}
Sep 07 22:36:56.190401 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:56.190277    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc" event={"ID":"eedebdef-2a83-45de-8b29-a8cd04283e0e","Type":"ContainerStarted","Data":"86531b0face23d9dc1caae34ca0a25fa159d7e0edab945c2d9f44df4e9115ca0"}
Sep 07 22:36:57.195379 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:57.195345    2582 generic.go:356] "Generic (PLEG): container finished" podID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerID="4055e9062f5bbd96b1291069c29960cb5316e63fb703ed37c395a6a5ce52e82c" exitCode=0
Sep 07 22:36:57.195760 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:57.195431    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc" event={"ID":"eedebdef-2a83-45de-8b29-a8cd04283e0e","Type":"ContainerDied","Data":"4055e9062f5bbd96b1291069c29960cb5316e63fb703ed37c395a6a5ce52e82c"}
Sep 07 22:36:58.200588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:58.200552    2582 generic.go:356] "Generic (PLEG): container finished" podID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerID="c06a0bf3b00c7f4250b5be10361057283bd80e94d1e32eaacb04761cb2b98ce3" exitCode=0
Sep 07 22:36:58.200964 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:58.200601    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc" event={"ID":"eedebdef-2a83-45de-8b29-a8cd04283e0e","Type":"ContainerDied","Data":"c06a0bf3b00c7f4250b5be10361057283bd80e94d1e32eaacb04761cb2b98ce3"}
Sep 07 22:36:59.329670 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.329642    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:36:59.494059 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.493949    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-util\") pod \"eedebdef-2a83-45de-8b29-a8cd04283e0e\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") "
Sep 07 22:36:59.494059 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.494001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hn8c2\" (UniqueName: \"kubernetes.io/projected/eedebdef-2a83-45de-8b29-a8cd04283e0e-kube-api-access-hn8c2\") pod \"eedebdef-2a83-45de-8b29-a8cd04283e0e\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") "
Sep 07 22:36:59.494059 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.494045    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-bundle\") pod \"eedebdef-2a83-45de-8b29-a8cd04283e0e\" (UID: \"eedebdef-2a83-45de-8b29-a8cd04283e0e\") "
Sep 07 22:36:59.494810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.494785    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-bundle" (OuterVolumeSpecName: "bundle") pod "eedebdef-2a83-45de-8b29-a8cd04283e0e" (UID: "eedebdef-2a83-45de-8b29-a8cd04283e0e"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:36:59.496314 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.496261    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eedebdef-2a83-45de-8b29-a8cd04283e0e-kube-api-access-hn8c2" (OuterVolumeSpecName: "kube-api-access-hn8c2") pod "eedebdef-2a83-45de-8b29-a8cd04283e0e" (UID: "eedebdef-2a83-45de-8b29-a8cd04283e0e"). InnerVolumeSpecName "kube-api-access-hn8c2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:36:59.499376 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.499356    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-util" (OuterVolumeSpecName: "util") pod "eedebdef-2a83-45de-8b29-a8cd04283e0e" (UID: "eedebdef-2a83-45de-8b29-a8cd04283e0e"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:36:59.595269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.595227    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-bundle\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:59.595269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.595260    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/eedebdef-2a83-45de-8b29-a8cd04283e0e-util\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:36:59.595269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:36:59.595270    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hn8c2\" (UniqueName: \"kubernetes.io/projected/eedebdef-2a83-45de-8b29-a8cd04283e0e-kube-api-access-hn8c2\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:37:00.208732 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:00.208695    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc" event={"ID":"eedebdef-2a83-45de-8b29-a8cd04283e0e","Type":"ContainerDied","Data":"86531b0face23d9dc1caae34ca0a25fa159d7e0edab945c2d9f44df4e9115ca0"}
Sep 07 22:37:00.208732 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:00.208730    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796afw9qqc"
Sep 07 22:37:00.208928 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:00.208748    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="86531b0face23d9dc1caae34ca0a25fa159d7e0edab945c2d9f44df4e9115ca0"
Sep 07 22:37:09.464916 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.464881    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"]
Sep 07 22:37:09.465436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.465130    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerName="pull"
Sep 07 22:37:09.465436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.465141    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerName="pull"
Sep 07 22:37:09.465436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.465150    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerName="extract"
Sep 07 22:37:09.465436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.465155    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerName="extract"
Sep 07 22:37:09.465436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.465162    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerName="util"
Sep 07 22:37:09.465436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.465169    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerName="util"
Sep 07 22:37:09.465436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.465211    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="eedebdef-2a83-45de-8b29-a8cd04283e0e" containerName="extract"
Sep 07 22:37:09.467134 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.467117    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.469911 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.469882    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 22:37:09.470711 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.470695    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 22:37:09.470777 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.470699    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rl8kx\""
Sep 07 22:37:09.477465 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.477439    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"]
Sep 07 22:37:09.660258 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.660223    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"]
Sep 07 22:37:09.662304 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.662272    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.665017 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.664988    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 07 22:37:09.665164 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.665074    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wdhlh\""
Sep 07 22:37:09.665164 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.665104    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 07 22:37:09.665572 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.665504    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.665572 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.665548    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.665711 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.665629    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr7dw\" (UniqueName: \"kubernetes.io/projected/8c26e010-6344-4851-a4b9-a409a3d8c3a1-kube-api-access-tr7dw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.665751 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.665732    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 07 22:37:09.665791 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.665769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 07 22:37:09.680326 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.680278    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"]
Sep 07 22:37:09.766669 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766579    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3bf17e-001d-4840-afff-e486c5503ba1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.766669 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf17e-001d-4840-afff-e486c5503ba1-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.766669 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766661    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr7dw\" (UniqueName: \"kubernetes.io/projected/8c26e010-6344-4851-a4b9-a409a3d8c3a1-kube-api-access-tr7dw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.766895 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766710    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf17e-001d-4840-afff-e486c5503ba1-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.766895 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766740    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzpx7\" (UniqueName: \"kubernetes.io/projected/7b3bf17e-001d-4840-afff-e486c5503ba1-kube-api-access-xzpx7\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.766895 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3bf17e-001d-4840-afff-e486c5503ba1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.766895 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766863    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.767043 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.766895    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.767224 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.767210    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-util\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.767272 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.767250    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-bundle\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.774560 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.774530    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr7dw\" (UniqueName: \"kubernetes.io/projected/8c26e010-6344-4851-a4b9-a409a3d8c3a1-kube-api-access-tr7dw\") pod \"8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") " pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.776143 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.776111    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:09.867978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.867938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3bf17e-001d-4840-afff-e486c5503ba1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.868230 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.867985    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf17e-001d-4840-afff-e486c5503ba1-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.868230 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.868033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf17e-001d-4840-afff-e486c5503ba1-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.868230 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.868060    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzpx7\" (UniqueName: \"kubernetes.io/projected/7b3bf17e-001d-4840-afff-e486c5503ba1-kube-api-access-xzpx7\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.868230 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.868092    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3bf17e-001d-4840-afff-e486c5503ba1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.868572 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.868546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf17e-001d-4840-afff-e486c5503ba1-manifests\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.870570 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.870539    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/7b3bf17e-001d-4840-afff-e486c5503ba1-tmp\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.870702 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.870682    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/7b3bf17e-001d-4840-afff-e486c5503ba1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.870875 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.870855    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b3bf17e-001d-4840-afff-e486c5503ba1-webhook-cert\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.875674 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.875653    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzpx7\" (UniqueName: \"kubernetes.io/projected/7b3bf17e-001d-4840-afff-e486c5503ba1-kube-api-access-xzpx7\") pod \"opendatahub-operator-controller-manager-7d648d744d-v2k2x\" (UID: \"7b3bf17e-001d-4840-afff-e486c5503ba1\") " pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:09.972352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:09.972309    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:10.049563 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.049525    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"]
Sep 07 22:37:10.051120 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:37:10.051084    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c26e010_6344_4851_a4b9_a409a3d8c3a1.slice/crio-e4f9898d49a44f32149b02538058cde7ca111efe405d80c2afe131fdadc6d93c WatchSource:0}: Error finding container e4f9898d49a44f32149b02538058cde7ca111efe405d80c2afe131fdadc6d93c: Status 404 returned error can't find the container with id e4f9898d49a44f32149b02538058cde7ca111efe405d80c2afe131fdadc6d93c
Sep 07 22:37:10.247794 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.247756    2582 generic.go:356] "Generic (PLEG): container finished" podID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerID="5f41aed651fe054f541ce30e90fc8d27c105b06809b616c62172775d05042fb1" exitCode=0
Sep 07 22:37:10.247942 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.247840    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf" event={"ID":"8c26e010-6344-4851-a4b9-a409a3d8c3a1","Type":"ContainerDied","Data":"5f41aed651fe054f541ce30e90fc8d27c105b06809b616c62172775d05042fb1"}
Sep 07 22:37:10.247942 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.247900    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf" event={"ID":"8c26e010-6344-4851-a4b9-a409a3d8c3a1","Type":"ContainerStarted","Data":"e4f9898d49a44f32149b02538058cde7ca111efe405d80c2afe131fdadc6d93c"}
Sep 07 22:37:10.252758 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.252733    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"]
Sep 07 22:37:10.254979 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:37:10.254957    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b3bf17e_001d_4840_afff_e486c5503ba1.slice/crio-c7effee4a53b7a445c5ee118e8e2b8198f448ac144c4b678b78ba99ad36a9490 WatchSource:0}: Error finding container c7effee4a53b7a445c5ee118e8e2b8198f448ac144c4b678b78ba99ad36a9490: Status 404 returned error can't find the container with id c7effee4a53b7a445c5ee118e8e2b8198f448ac144c4b678b78ba99ad36a9490
Sep 07 22:37:10.304014 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.303926    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"]
Sep 07 22:37:10.306176 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.306157    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.308566 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.308535    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 07 22:37:10.308566 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.308551    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 07 22:37:10.308759 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.308737    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 07 22:37:10.308843 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.308827    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-9s5f6\""
Sep 07 22:37:10.316252 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.316228    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"]
Sep 07 22:37:10.372821 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.372776    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dee80fb7-723a-406e-b9ed-fd3bc99a0335-metrics-cert\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.372821 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.372823    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee80fb7-723a-406e-b9ed-fd3bc99a0335-cert\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.373039 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.372846    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9ncf\" (UniqueName: \"kubernetes.io/projected/dee80fb7-723a-406e-b9ed-fd3bc99a0335-kube-api-access-c9ncf\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.373039 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.372915    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dee80fb7-723a-406e-b9ed-fd3bc99a0335-manager-config\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.474182 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.474141    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dee80fb7-723a-406e-b9ed-fd3bc99a0335-manager-config\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.474675 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.474210    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dee80fb7-723a-406e-b9ed-fd3bc99a0335-metrics-cert\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.474675 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.474232    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee80fb7-723a-406e-b9ed-fd3bc99a0335-cert\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.474675 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.474252    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9ncf\" (UniqueName: \"kubernetes.io/projected/dee80fb7-723a-406e-b9ed-fd3bc99a0335-kube-api-access-c9ncf\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.474879 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.474858    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/dee80fb7-723a-406e-b9ed-fd3bc99a0335-manager-config\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.476729 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.476704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/dee80fb7-723a-406e-b9ed-fd3bc99a0335-cert\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.476825 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.476744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/dee80fb7-723a-406e-b9ed-fd3bc99a0335-metrics-cert\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.482653 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.482631    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9ncf\" (UniqueName: \"kubernetes.io/projected/dee80fb7-723a-406e-b9ed-fd3bc99a0335-kube-api-access-c9ncf\") pod \"lws-controller-manager-575c877bb5-hrp27\" (UID: \"dee80fb7-723a-406e-b9ed-fd3bc99a0335\") " pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.616362 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.616304    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:10.896820 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:10.896724    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"]
Sep 07 22:37:10.900009 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:37:10.899981    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddee80fb7_723a_406e_b9ed_fd3bc99a0335.slice/crio-362e0e4fc653ea2b9c5d273aa26e34f09db8670fa73fdfd0ab31ba6c27dd8d61 WatchSource:0}: Error finding container 362e0e4fc653ea2b9c5d273aa26e34f09db8670fa73fdfd0ab31ba6c27dd8d61: Status 404 returned error can't find the container with id 362e0e4fc653ea2b9c5d273aa26e34f09db8670fa73fdfd0ab31ba6c27dd8d61
Sep 07 22:37:11.257994 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:11.257905    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27" event={"ID":"dee80fb7-723a-406e-b9ed-fd3bc99a0335","Type":"ContainerStarted","Data":"362e0e4fc653ea2b9c5d273aa26e34f09db8670fa73fdfd0ab31ba6c27dd8d61"}
Sep 07 22:37:11.259704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:11.259624    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x" event={"ID":"7b3bf17e-001d-4840-afff-e486c5503ba1","Type":"ContainerStarted","Data":"c7effee4a53b7a445c5ee118e8e2b8198f448ac144c4b678b78ba99ad36a9490"}
Sep 07 22:37:11.262166 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:11.262128    2582 generic.go:356] "Generic (PLEG): container finished" podID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerID="9a90c053055cc31663f1b44b68900bd5ad47a8c703eca6351b7356ea9914b6f1" exitCode=0
Sep 07 22:37:11.262306 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:11.262229    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf" event={"ID":"8c26e010-6344-4851-a4b9-a409a3d8c3a1","Type":"ContainerDied","Data":"9a90c053055cc31663f1b44b68900bd5ad47a8c703eca6351b7356ea9914b6f1"}
Sep 07 22:37:12.269966 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:12.269922    2582 generic.go:356] "Generic (PLEG): container finished" podID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerID="e7320beddab490559a72609d5808b20d945e0925c2f713fed6f1fabb9fcdc916" exitCode=0
Sep 07 22:37:12.270403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:12.269975    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf" event={"ID":"8c26e010-6344-4851-a4b9-a409a3d8c3a1","Type":"ContainerDied","Data":"e7320beddab490559a72609d5808b20d945e0925c2f713fed6f1fabb9fcdc916"}
Sep 07 22:37:13.598329 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.598303    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:13.701846 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.701814    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-bundle\") pod \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") "
Sep 07 22:37:13.702023 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.701876    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-util\") pod \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") "
Sep 07 22:37:13.702023 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.701928    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tr7dw\" (UniqueName: \"kubernetes.io/projected/8c26e010-6344-4851-a4b9-a409a3d8c3a1-kube-api-access-tr7dw\") pod \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\" (UID: \"8c26e010-6344-4851-a4b9-a409a3d8c3a1\") "
Sep 07 22:37:13.702652 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.702622    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-bundle" (OuterVolumeSpecName: "bundle") pod "8c26e010-6344-4851-a4b9-a409a3d8c3a1" (UID: "8c26e010-6344-4851-a4b9-a409a3d8c3a1"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:37:13.704067 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.704031    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c26e010-6344-4851-a4b9-a409a3d8c3a1-kube-api-access-tr7dw" (OuterVolumeSpecName: "kube-api-access-tr7dw") pod "8c26e010-6344-4851-a4b9-a409a3d8c3a1" (UID: "8c26e010-6344-4851-a4b9-a409a3d8c3a1"). InnerVolumeSpecName "kube-api-access-tr7dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:37:13.707766 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.707723    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-util" (OuterVolumeSpecName: "util") pod "8c26e010-6344-4851-a4b9-a409a3d8c3a1" (UID: "8c26e010-6344-4851-a4b9-a409a3d8c3a1"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:37:13.802932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.802885    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr7dw\" (UniqueName: \"kubernetes.io/projected/8c26e010-6344-4851-a4b9-a409a3d8c3a1-kube-api-access-tr7dw\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:37:13.802932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.802920    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-bundle\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:37:13.802932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.802931    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/8c26e010-6344-4851-a4b9-a409a3d8c3a1-util\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:37:13.922053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:13.921973    2582 kubelet.go:1648] "Image garbage collection succeeded"
Sep 07 22:37:14.280612 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:14.280505    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27" event={"ID":"dee80fb7-723a-406e-b9ed-fd3bc99a0335","Type":"ContainerStarted","Data":"c3045acfbe26036dd1fce56a176a7356ae33fd56ed063792ad3aede78f5820eb"}
Sep 07 22:37:14.280801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:14.280625    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:14.282121 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:14.282099    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x" event={"ID":"7b3bf17e-001d-4840-afff-e486c5503ba1","Type":"ContainerStarted","Data":"343ebf05aa9fe51beae094e34ec526eb6c23bc57a987518e01386c1a4f50f110"}
Sep 07 22:37:14.284031 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:14.284014    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf"
Sep 07 22:37:14.284119 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:14.284013    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/8e0ea019126e0c85f070b2014e3108eef38b84ccf513200a96dc7d56b05ljrf" event={"ID":"8c26e010-6344-4851-a4b9-a409a3d8c3a1","Type":"ContainerDied","Data":"e4f9898d49a44f32149b02538058cde7ca111efe405d80c2afe131fdadc6d93c"}
Sep 07 22:37:14.284159 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:14.284125    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4f9898d49a44f32149b02538058cde7ca111efe405d80c2afe131fdadc6d93c"
Sep 07 22:37:14.296432 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:14.296376    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27" podStartSLOduration=1.593067743 podStartE2EDuration="4.296358302s" totalImagesPullingTime="2.703290559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:14.295352145 +0000 UTC m=+300.761509130" watchObservedRunningTime="2026-09-07 22:37:14.296358302 +0000 UTC m=+300.762515297"
Sep 07 22:37:15.289830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:15.289793    2582 generic.go:356] "Generic (PLEG): container finished" podID="7b3bf17e-001d-4840-afff-e486c5503ba1" containerID="343ebf05aa9fe51beae094e34ec526eb6c23bc57a987518e01386c1a4f50f110" exitCode=0
Sep 07 22:37:15.290207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:15.289866    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x" event={"ID":"7b3bf17e-001d-4840-afff-e486c5503ba1","Type":"ContainerDied","Data":"343ebf05aa9fe51beae094e34ec526eb6c23bc57a987518e01386c1a4f50f110"}
Sep 07 22:37:15.290376 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:15.290361    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:37:16.296414 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:16.296369    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x" event={"ID":"7b3bf17e-001d-4840-afff-e486c5503ba1","Type":"ContainerStarted","Data":"c7d3661d0cd107a5fdbe4ba30cfa8ba557b9a453f1064775b907711e00475236"}
Sep 07 22:37:16.296813 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:16.296523    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:16.316872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:16.316820    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x" podStartSLOduration=2.369924943 podStartE2EDuration="7.316802562s" totalImagesPullingTime="3.814883139s" totalInitContainerRuntime="1.13199448s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:09 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:16.314418558 +0000 UTC m=+302.780575579" watchObservedRunningTime="2026-09-07 22:37:16.316802562 +0000 UTC m=+302.782959558"
Sep 07 22:37:25.292639 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:25.292600    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-575c877bb5-hrp27"
Sep 07 22:37:27.302913 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:27.302879    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7d648d744d-v2k2x"
Sep 07 22:37:36.567129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567091    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"]
Sep 07 22:37:36.567523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567387    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerName="pull"
Sep 07 22:37:36.567523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567400    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerName="pull"
Sep 07 22:37:36.567523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567411    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerName="extract"
Sep 07 22:37:36.567523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567417    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerName="extract"
Sep 07 22:37:36.567523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567433    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerName="util"
Sep 07 22:37:36.567523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567439    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerName="util"
Sep 07 22:37:36.567523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.567481    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c26e010-6344-4851-a4b9-a409a3d8c3a1" containerName="extract"
Sep 07 22:37:36.570189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.570172    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.572746 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.572720    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 07 22:37:36.572746 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.572732    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-lbjfl\""
Sep 07 22:37:36.572963 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.572732    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 07 22:37:36.572963 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.572761    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 07 22:37:36.580937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.580914    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"]
Sep 07 22:37:36.673932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.673875    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.673932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.673939    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.674146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.673980    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nln97\" (UniqueName: \"kubernetes.io/projected/f36ad099-80fa-4c53-a72a-9513fbf459e0-kube-api-access-nln97\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.674146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.674007    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.674146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.674036    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.674146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.674064    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.674146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.674081    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ad099-80fa-4c53-a72a-9513fbf459e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.674146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.674117    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.674146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.674137    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775354 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775307    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775354 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775357    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775586 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775411    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775586 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775445    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775586 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775478    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nln97\" (UniqueName: \"kubernetes.io/projected/f36ad099-80fa-4c53-a72a-9513fbf459e0-kube-api-access-nln97\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775586 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775513    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775586 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775548    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775828 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775599    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775828 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775627    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ad099-80fa-4c53-a72a-9513fbf459e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775828 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775777    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775982 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775926    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.775982 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.775955    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.776088 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.776001    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.776346 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.776325    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f36ad099-80fa-4c53-a72a-9513fbf459e0-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.777819 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.777795    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.778141 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.778120    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.784538 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.784516    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f36ad099-80fa-4c53-a72a-9513fbf459e0-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.784757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.784736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nln97\" (UniqueName: \"kubernetes.io/projected/f36ad099-80fa-4c53-a72a-9513fbf459e0-kube-api-access-nln97\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn\" (UID: \"f36ad099-80fa-4c53-a72a-9513fbf459e0\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:36.880839 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:36.880750    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:37.155258 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:37.155232    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"]
Sep 07 22:37:37.157564 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:37:37.157526    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf36ad099_80fa_4c53_a72a_9513fbf459e0.slice/crio-2bf31d488a536d02f4bca5692c9f69ad29a7d272b18b11ab625de2e1d1601816 WatchSource:0}: Error finding container 2bf31d488a536d02f4bca5692c9f69ad29a7d272b18b11ab625de2e1d1601816: Status 404 returned error can't find the container with id 2bf31d488a536d02f4bca5692c9f69ad29a7d272b18b11ab625de2e1d1601816
Sep 07 22:37:37.383270 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:37.383233    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn" event={"ID":"f36ad099-80fa-4c53-a72a-9513fbf459e0","Type":"ContainerStarted","Data":"2bf31d488a536d02f4bca5692c9f69ad29a7d272b18b11ab625de2e1d1601816"}
Sep 07 22:37:40.344740 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:40.344699    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:37:40.345037 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:40.344780    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:37:40.345037 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:40.344807    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:37:41.403726 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:41.403689    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn" event={"ID":"f36ad099-80fa-4c53-a72a-9513fbf459e0","Type":"ContainerStarted","Data":"05549806cdf19aaf041ed6dcd90e5e4cb96c1f782fb722ecc45b52f35b98e0eb"}
Sep 07 22:37:41.423399 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:41.423340    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn" podStartSLOduration=2.238567986 podStartE2EDuration="5.423323736s" totalImagesPullingTime="3.18475575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:41.421696111 +0000 UTC m=+327.887853106" watchObservedRunningTime="2026-09-07 22:37:41.423323736 +0000 UTC m=+327.889480730"
Sep 07 22:37:41.881877 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:41.881833    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:41.886269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:41.886241    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:42.408349 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:42.408313    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:42.409405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:42.409384    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn"
Sep 07 22:37:49.285645 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.285611    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"]
Sep 07 22:37:49.289101 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.289073    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.291587 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.291560    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 07 22:37:49.291705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.291625    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-b7dfh\""
Sep 07 22:37:49.298084 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.298059    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"]
Sep 07 22:37:49.379939 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.379898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt26q\" (UniqueName: \"kubernetes.io/projected/8a7dacde-558e-48e5-b9e0-cea7b4d363c0-kube-api-access-qt26q\") pod \"llmisvc-controller-manager-777c6bd7d5-7lcvs\" (UID: \"8a7dacde-558e-48e5-b9e0-cea7b4d363c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.380113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.380013    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7dacde-558e-48e5-b9e0-cea7b4d363c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7lcvs\" (UID: \"8a7dacde-558e-48e5-b9e0-cea7b4d363c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.480680 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.480637    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt26q\" (UniqueName: \"kubernetes.io/projected/8a7dacde-558e-48e5-b9e0-cea7b4d363c0-kube-api-access-qt26q\") pod \"llmisvc-controller-manager-777c6bd7d5-7lcvs\" (UID: \"8a7dacde-558e-48e5-b9e0-cea7b4d363c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.480859 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.480712    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7dacde-558e-48e5-b9e0-cea7b4d363c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7lcvs\" (UID: \"8a7dacde-558e-48e5-b9e0-cea7b4d363c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.483166 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.483143    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8a7dacde-558e-48e5-b9e0-cea7b4d363c0-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-7lcvs\" (UID: \"8a7dacde-558e-48e5-b9e0-cea7b4d363c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.494347 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.494317    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt26q\" (UniqueName: \"kubernetes.io/projected/8a7dacde-558e-48e5-b9e0-cea7b4d363c0-kube-api-access-qt26q\") pod \"llmisvc-controller-manager-777c6bd7d5-7lcvs\" (UID: \"8a7dacde-558e-48e5-b9e0-cea7b4d363c0\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.600264 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.600223    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:49.869716 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:49.869679    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"]
Sep 07 22:37:49.871700 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:37:49.871665    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod8a7dacde_558e_48e5_b9e0_cea7b4d363c0.slice/crio-8a690489ecb0fac179515b861ae532918914a4e2db39a57796f328ab3cad4d04 WatchSource:0}: Error finding container 8a690489ecb0fac179515b861ae532918914a4e2db39a57796f328ab3cad4d04: Status 404 returned error can't find the container with id 8a690489ecb0fac179515b861ae532918914a4e2db39a57796f328ab3cad4d04
Sep 07 22:37:50.442538 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:50.442493    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs" event={"ID":"8a7dacde-558e-48e5-b9e0-cea7b4d363c0","Type":"ContainerStarted","Data":"8a690489ecb0fac179515b861ae532918914a4e2db39a57796f328ab3cad4d04"}
Sep 07 22:37:53.457777 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:53.457741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs" event={"ID":"8a7dacde-558e-48e5-b9e0-cea7b4d363c0","Type":"ContainerStarted","Data":"d077d2c8991edcca81b585fd4b7e00d046e2269d24eae37d7dd0c41e27044f7e"}
Sep 07 22:37:53.458181 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:53.457871    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:37:53.473296 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:37:53.473235    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs" podStartSLOduration=1.478202826 podStartE2EDuration="4.473218111s" totalImagesPullingTime="2.995015285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:37:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:37:53.470894671 +0000 UTC m=+339.937051678" watchObservedRunningTime="2026-09-07 22:37:53.473218111 +0000 UTC m=+339.939375106"
Sep 07 22:38:24.465062 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:24.465020    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-7lcvs"
Sep 07 22:38:44.094821 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.094779    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"]
Sep 07 22:38:44.098107 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.098088    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.100488 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.100466    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Sep 07 22:38:44.101269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.101253    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Sep 07 22:38:44.101330 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.101274    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-rl8kx\""
Sep 07 22:38:44.105958 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.105930    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"]
Sep 07 22:38:44.128344 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.128299    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.128515 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.128354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x865h\" (UniqueName: \"kubernetes.io/projected/fe29dbce-17d5-4948-9682-10800bf07164-kube-api-access-x865h\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.128515 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.128399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.229138 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.229085    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.229138 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.229142    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x865h\" (UniqueName: \"kubernetes.io/projected/fe29dbce-17d5-4948-9682-10800bf07164-kube-api-access-x865h\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.229447 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.229198    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.229615 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.229590    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-util\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.229679 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.229611    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-bundle\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.236974 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.236947    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x865h\" (UniqueName: \"kubernetes.io/projected/fe29dbce-17d5-4948-9682-10800bf07164-kube-api-access-x865h\") pod \"aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") " pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.407470 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.407364    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:44.678720 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:44.678696    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"]
Sep 07 22:38:44.681130 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:38:44.681101    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe29dbce_17d5_4948_9682_10800bf07164.slice/crio-0cd663c7d9962a1ffc2fe771356ebd279db61cc846fe660efd67ad4fb824dc0e WatchSource:0}: Error finding container 0cd663c7d9962a1ffc2fe771356ebd279db61cc846fe660efd67ad4fb824dc0e: Status 404 returned error can't find the container with id 0cd663c7d9962a1ffc2fe771356ebd279db61cc846fe660efd67ad4fb824dc0e
Sep 07 22:38:45.677327 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:45.677266    2582 generic.go:356] "Generic (PLEG): container finished" podID="fe29dbce-17d5-4948-9682-10800bf07164" containerID="e4eaa59713cd3c562e40f6975fcaf9ad43eb94f74548215b5149f5af73692bc9" exitCode=0
Sep 07 22:38:45.677722 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:45.677352    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2" event={"ID":"fe29dbce-17d5-4948-9682-10800bf07164","Type":"ContainerDied","Data":"e4eaa59713cd3c562e40f6975fcaf9ad43eb94f74548215b5149f5af73692bc9"}
Sep 07 22:38:45.677722 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:45.677405    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2" event={"ID":"fe29dbce-17d5-4948-9682-10800bf07164","Type":"ContainerStarted","Data":"0cd663c7d9962a1ffc2fe771356ebd279db61cc846fe660efd67ad4fb824dc0e"}
Sep 07 22:38:46.686710 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:46.686676    2582 generic.go:356] "Generic (PLEG): container finished" podID="fe29dbce-17d5-4948-9682-10800bf07164" containerID="44890085da3147ce6c17a54ee5f70c1611b360dd8a051aa7f5eba5a572057a33" exitCode=0
Sep 07 22:38:46.687113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:46.686759    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2" event={"ID":"fe29dbce-17d5-4948-9682-10800bf07164","Type":"ContainerDied","Data":"44890085da3147ce6c17a54ee5f70c1611b360dd8a051aa7f5eba5a572057a33"}
Sep 07 22:38:47.693806 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:47.693769    2582 generic.go:356] "Generic (PLEG): container finished" podID="fe29dbce-17d5-4948-9682-10800bf07164" containerID="c74053c9b0def2388dca728df6366000ba8e13f889ffa8298f9cac7c5c1fb53a" exitCode=0
Sep 07 22:38:47.694251 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:47.693848    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2" event={"ID":"fe29dbce-17d5-4948-9682-10800bf07164","Type":"ContainerDied","Data":"c74053c9b0def2388dca728df6366000ba8e13f889ffa8298f9cac7c5c1fb53a"}
Sep 07 22:38:48.823113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.823083    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:48.866668 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.866629    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x865h\" (UniqueName: \"kubernetes.io/projected/fe29dbce-17d5-4948-9682-10800bf07164-kube-api-access-x865h\") pod \"fe29dbce-17d5-4948-9682-10800bf07164\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") "
Sep 07 22:38:48.866855 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.866678    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-bundle\") pod \"fe29dbce-17d5-4948-9682-10800bf07164\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") "
Sep 07 22:38:48.866855 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.866705    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-util\") pod \"fe29dbce-17d5-4948-9682-10800bf07164\" (UID: \"fe29dbce-17d5-4948-9682-10800bf07164\") "
Sep 07 22:38:48.867160 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.867131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-bundle" (OuterVolumeSpecName: "bundle") pod "fe29dbce-17d5-4948-9682-10800bf07164" (UID: "fe29dbce-17d5-4948-9682-10800bf07164"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:48.868808 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.868780    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe29dbce-17d5-4948-9682-10800bf07164-kube-api-access-x865h" (OuterVolumeSpecName: "kube-api-access-x865h") pod "fe29dbce-17d5-4948-9682-10800bf07164" (UID: "fe29dbce-17d5-4948-9682-10800bf07164"). InnerVolumeSpecName "kube-api-access-x865h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:38:48.871881 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.871853    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-util" (OuterVolumeSpecName: "util") pod "fe29dbce-17d5-4948-9682-10800bf07164" (UID: "fe29dbce-17d5-4948-9682-10800bf07164"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:38:48.967539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.967443    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x865h\" (UniqueName: \"kubernetes.io/projected/fe29dbce-17d5-4948-9682-10800bf07164-kube-api-access-x865h\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:38:48.967539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.967479    2582 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-bundle\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:38:48.967539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:48.967490    2582 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fe29dbce-17d5-4948-9682-10800bf07164-util\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:38:49.705028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:49.704992    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2" event={"ID":"fe29dbce-17d5-4948-9682-10800bf07164","Type":"ContainerDied","Data":"0cd663c7d9962a1ffc2fe771356ebd279db61cc846fe660efd67ad4fb824dc0e"}
Sep 07 22:38:49.705028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:49.705014    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/aa4484d327db8038362a08ac827b2b7d95fdfb52c243a86a3fcc5d78bb7vtp2"
Sep 07 22:38:49.705255 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:49.705043    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0cd663c7d9962a1ffc2fe771356ebd279db61cc846fe660efd67ad4fb824dc0e"
Sep 07 22:38:59.010427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010383    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"]
Sep 07 22:38:59.010910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010670    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe29dbce-17d5-4948-9682-10800bf07164" containerName="pull"
Sep 07 22:38:59.010910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010682    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe29dbce-17d5-4948-9682-10800bf07164" containerName="pull"
Sep 07 22:38:59.010910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010691    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe29dbce-17d5-4948-9682-10800bf07164" containerName="extract"
Sep 07 22:38:59.010910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010696    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe29dbce-17d5-4948-9682-10800bf07164" containerName="extract"
Sep 07 22:38:59.010910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010707    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe29dbce-17d5-4948-9682-10800bf07164" containerName="util"
Sep 07 22:38:59.010910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010712    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe29dbce-17d5-4948-9682-10800bf07164" containerName="util"
Sep 07 22:38:59.010910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.010761    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe29dbce-17d5-4948-9682-10800bf07164" containerName="extract"
Sep 07 22:38:59.014081 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.014063    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"
Sep 07 22:38:59.016641 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.016614    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 07 22:38:59.016801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.016698    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 07 22:38:59.016801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.016739    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 07 22:38:59.016909 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.016746    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-bzbsm\""
Sep 07 22:38:59.023982 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.023944    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"]
Sep 07 22:38:59.152889 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.152844    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tqwn8\" (UniqueName: \"kubernetes.io/projected/b4c23af5-92c5-43a0-b236-7c53ad497b9f-kube-api-access-tqwn8\") pod \"dns-operator-controller-manager-5985564776-jbdcs\" (UID: \"b4c23af5-92c5-43a0-b236-7c53ad497b9f\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"
Sep 07 22:38:59.253435 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.253390    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tqwn8\" (UniqueName: \"kubernetes.io/projected/b4c23af5-92c5-43a0-b236-7c53ad497b9f-kube-api-access-tqwn8\") pod \"dns-operator-controller-manager-5985564776-jbdcs\" (UID: \"b4c23af5-92c5-43a0-b236-7c53ad497b9f\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"
Sep 07 22:38:59.270008 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.269942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tqwn8\" (UniqueName: \"kubernetes.io/projected/b4c23af5-92c5-43a0-b236-7c53ad497b9f-kube-api-access-tqwn8\") pod \"dns-operator-controller-manager-5985564776-jbdcs\" (UID: \"b4c23af5-92c5-43a0-b236-7c53ad497b9f\") " pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"
Sep 07 22:38:59.325055 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.325009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"
Sep 07 22:38:59.597228 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.597200    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"]
Sep 07 22:38:59.599753 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:38:59.599728    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4c23af5_92c5_43a0_b236_7c53ad497b9f.slice/crio-6ca13c00722e5eaf2298632b6c2316702c6fc94ed808ac139232d2dd7e9e2316 WatchSource:0}: Error finding container 6ca13c00722e5eaf2298632b6c2316702c6fc94ed808ac139232d2dd7e9e2316: Status 404 returned error can't find the container with id 6ca13c00722e5eaf2298632b6c2316702c6fc94ed808ac139232d2dd7e9e2316
Sep 07 22:38:59.749881 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:38:59.749841    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs" event={"ID":"b4c23af5-92c5-43a0-b236-7c53ad497b9f","Type":"ContainerStarted","Data":"6ca13c00722e5eaf2298632b6c2316702c6fc94ed808ac139232d2dd7e9e2316"}
Sep 07 22:39:01.397507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.397468    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"]
Sep 07 22:39:01.406672 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.406639    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:01.409778 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.409754    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-2kqgq\""
Sep 07 22:39:01.416461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.416435    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"]
Sep 07 22:39:01.472657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.472617    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmrct\" (UniqueName: \"kubernetes.io/projected/90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3-kube-api-access-qmrct\") pod \"limitador-operator-controller-manager-6c549b6d85-c45sv\" (UID: \"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:01.574459 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.574418    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmrct\" (UniqueName: \"kubernetes.io/projected/90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3-kube-api-access-qmrct\") pod \"limitador-operator-controller-manager-6c549b6d85-c45sv\" (UID: \"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:01.586072 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.586031    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmrct\" (UniqueName: \"kubernetes.io/projected/90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3-kube-api-access-qmrct\") pod \"limitador-operator-controller-manager-6c549b6d85-c45sv\" (UID: \"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3\") " pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:01.719971 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:01.719875    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:02.390962 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:02.390753    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"]
Sep 07 22:39:02.393625 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:39:02.393594    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90b2a1f9_1632_4cd4_b240_d6c6ef7d0aa3.slice/crio-c3a6197cedf7e7f9b4be55f58c3d262a2176fed539f1ce27cb217146a381597e WatchSource:0}: Error finding container c3a6197cedf7e7f9b4be55f58c3d262a2176fed539f1ce27cb217146a381597e: Status 404 returned error can't find the container with id c3a6197cedf7e7f9b4be55f58c3d262a2176fed539f1ce27cb217146a381597e
Sep 07 22:39:02.767053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:02.766962    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs" event={"ID":"b4c23af5-92c5-43a0-b236-7c53ad497b9f","Type":"ContainerStarted","Data":"6e418a8242adcf63ab67002b5d8c5b8d460d301e912c4856d23538607e1a47e7"}
Sep 07 22:39:02.767510 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:02.767054    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"
Sep 07 22:39:02.768266 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:02.768242    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv" event={"ID":"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3","Type":"ContainerStarted","Data":"c3a6197cedf7e7f9b4be55f58c3d262a2176fed539f1ce27cb217146a381597e"}
Sep 07 22:39:02.795893 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:02.795835    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs" podStartSLOduration=2.214298752 podStartE2EDuration="4.79581543s" totalImagesPullingTime="2.581516678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:38:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:02.795423195 +0000 UTC m=+409.261580190" watchObservedRunningTime="2026-09-07 22:39:02.79581543 +0000 UTC m=+409.261972425"
Sep 07 22:39:04.779125 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:04.779083    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv" event={"ID":"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3","Type":"ContainerStarted","Data":"2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba"}
Sep 07 22:39:04.779529 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:04.779220    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:04.801383 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:04.801333    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv" podStartSLOduration=1.54985793 podStartE2EDuration="3.801317768s" totalImagesPullingTime="2.251459838s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:04.799813405 +0000 UTC m=+411.265970401" watchObservedRunningTime="2026-09-07 22:39:04.801317768 +0000 UTC m=+411.267474764"
Sep 07 22:39:05.401883 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.401848    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-64bbf7646b-sfpt2"]
Sep 07 22:39:05.405022 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.405003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.408706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408521    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 07 22:39:05.408706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408556    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-lhcs8\""
Sep 07 22:39:05.408706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408573    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 07 22:39:05.408706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408582    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 07 22:39:05.408706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408638    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 07 22:39:05.408706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408654    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 07 22:39:05.408706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408672    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 07 22:39:05.409107 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.408913    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 07 22:39:05.412904 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.412883    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Sep 07 22:39:05.416860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.416836    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bbf7646b-sfpt2"]
Sep 07 22:39:05.511676 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.511619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-trusted-ca-bundle\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.511676 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.511672    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-oauth-serving-cert\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.511915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.511737    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8g5b\" (UniqueName: \"kubernetes.io/projected/19c511cb-0e33-4ab6-9071-9741b0feb41a-kube-api-access-k8g5b\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.511915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.511794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-config\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.511915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.511825    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-serving-cert\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.511915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.511876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-service-ca\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.511915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.511902    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-oauth-config\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613130 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613090    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8g5b\" (UniqueName: \"kubernetes.io/projected/19c511cb-0e33-4ab6-9071-9741b0feb41a-kube-api-access-k8g5b\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613139    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-config\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613164    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-serving-cert\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613184    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-service-ca\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613319    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-oauth-config\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613561 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613389    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-trusted-ca-bundle\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613561 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613421    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-oauth-serving-cert\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.613987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.613945    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-service-ca\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.614102 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.614027    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-config\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.614244 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.614223    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-oauth-serving-cert\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.614338 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.614322    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/19c511cb-0e33-4ab6-9071-9741b0feb41a-trusted-ca-bundle\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.615849 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.615826    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-serving-cert\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.615986 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.615828    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/19c511cb-0e33-4ab6-9071-9741b0feb41a-console-oauth-config\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.623892 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.623869    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8g5b\" (UniqueName: \"kubernetes.io/projected/19c511cb-0e33-4ab6-9071-9741b0feb41a-kube-api-access-k8g5b\") pod \"console-64bbf7646b-sfpt2\" (UID: \"19c511cb-0e33-4ab6-9071-9741b0feb41a\") " pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.714347 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.714211    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:05.987837 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:05.987806    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-64bbf7646b-sfpt2"]
Sep 07 22:39:05.990235 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:39:05.990209    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19c511cb_0e33_4ab6_9071_9741b0feb41a.slice/crio-eeb16fbe1de3de2078e9c6cc534d5b1d6efa3369b96ed024b42cd84ddea3acb2 WatchSource:0}: Error finding container eeb16fbe1de3de2078e9c6cc534d5b1d6efa3369b96ed024b42cd84ddea3acb2: Status 404 returned error can't find the container with id eeb16fbe1de3de2078e9c6cc534d5b1d6efa3369b96ed024b42cd84ddea3acb2
Sep 07 22:39:06.790603 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:06.790561    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bbf7646b-sfpt2" event={"ID":"19c511cb-0e33-4ab6-9071-9741b0feb41a","Type":"ContainerStarted","Data":"8396e326d95893394c81b8dbc92c776451c7878b4523f20e39224c4d94bce51e"}
Sep 07 22:39:06.790819 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:06.790619    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-64bbf7646b-sfpt2" event={"ID":"19c511cb-0e33-4ab6-9071-9741b0feb41a","Type":"ContainerStarted","Data":"eeb16fbe1de3de2078e9c6cc534d5b1d6efa3369b96ed024b42cd84ddea3acb2"}
Sep 07 22:39:06.808949 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:06.808893    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-64bbf7646b-sfpt2" podStartSLOduration=1.8088764689999999 podStartE2EDuration="1.808876469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:06.807445428 +0000 UTC m=+413.273602459" watchObservedRunningTime="2026-09-07 22:39:06.808876469 +0000 UTC m=+413.275033464"
Sep 07 22:39:07.480982 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.480945    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"]
Sep 07 22:39:07.483907 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.483885    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.486210 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.486182    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-nwtjv\""
Sep 07 22:39:07.486210 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.486195    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 07 22:39:07.486497 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.486480    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 07 22:39:07.491799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.491771    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"]
Sep 07 22:39:07.632135 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.632084    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hgh9\" (UniqueName: \"kubernetes.io/projected/76d16438-b115-4c83-803b-aa7fe8b53236-kube-api-access-8hgh9\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.632135 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.632145    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76d16438-b115-4c83-803b-aa7fe8b53236-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.632381 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.632214    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76d16438-b115-4c83-803b-aa7fe8b53236-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.733047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.732958    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76d16438-b115-4c83-803b-aa7fe8b53236-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.733047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.733035    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hgh9\" (UniqueName: \"kubernetes.io/projected/76d16438-b115-4c83-803b-aa7fe8b53236-kube-api-access-8hgh9\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.733254 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.733134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76d16438-b115-4c83-803b-aa7fe8b53236-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.733794 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.733768    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/76d16438-b115-4c83-803b-aa7fe8b53236-nginx-conf\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.735408 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.735390    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/76d16438-b115-4c83-803b-aa7fe8b53236-plugin-serving-cert\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.740758 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.740731    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hgh9\" (UniqueName: \"kubernetes.io/projected/76d16438-b115-4c83-803b-aa7fe8b53236-kube-api-access-8hgh9\") pod \"kuadrant-console-plugin-598f86468b-zn48v\" (UID: \"76d16438-b115-4c83-803b-aa7fe8b53236\") " pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:07.794515 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:07.794475    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"
Sep 07 22:39:08.063113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:08.063084    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v"]
Sep 07 22:39:08.065222 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:39:08.065192    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76d16438_b115_4c83_803b_aa7fe8b53236.slice/crio-2e781f90c9895d8e5854a9948c428161b90974839fdd4af4eb8038fe7d451195 WatchSource:0}: Error finding container 2e781f90c9895d8e5854a9948c428161b90974839fdd4af4eb8038fe7d451195: Status 404 returned error can't find the container with id 2e781f90c9895d8e5854a9948c428161b90974839fdd4af4eb8038fe7d451195
Sep 07 22:39:08.801486 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:08.801450    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v" event={"ID":"76d16438-b115-4c83-803b-aa7fe8b53236","Type":"ContainerStarted","Data":"2e781f90c9895d8e5854a9948c428161b90974839fdd4af4eb8038fe7d451195"}
Sep 07 22:39:13.775441 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:13.775406    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-5985564776-jbdcs"
Sep 07 22:39:13.827888 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:13.827849    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v" event={"ID":"76d16438-b115-4c83-803b-aa7fe8b53236","Type":"ContainerStarted","Data":"716a9979366c689861f2cb920d42181e3a826f48e8743405c2388aaf0f2f6879"}
Sep 07 22:39:13.843011 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:13.842958    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-598f86468b-zn48v" podStartSLOduration=2.149209674 podStartE2EDuration="6.842941099s" totalImagesPullingTime="4.693731425s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:13.841541473 +0000 UTC m=+420.307698469" watchObservedRunningTime="2026-09-07 22:39:13.842941099 +0000 UTC m=+420.309098113"
Sep 07 22:39:15.714657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:15.714618    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:15.715114 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:15.714681    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:15.719450 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:15.719423    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:15.786352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:15.786318    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:15.840916 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:15.840890    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-64bbf7646b-sfpt2"
Sep 07 22:39:18.819131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:18.819090    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"]
Sep 07 22:39:18.819554 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:18.819310    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" containerName="manager" containerID="cri-o://2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba" gracePeriod=2
Sep 07 22:39:18.836469 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:18.836437    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"]
Sep 07 22:39:18.852119 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:39:18.852067    2582 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"limitador-operator-controller-manager-6c549b6d85-c45sv\": pods \"limitador-operator-controller-manager-6c549b6d85-c45sv\" not found" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:19.064670 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.064637    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:19.066746 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:39:19.066717    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-c45sv\" is forbidden: User \"system:node:ip-10-0-137-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-80.ec2.internal' and this object" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:19.238170 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.238085    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qmrct\" (UniqueName: \"kubernetes.io/projected/90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3-kube-api-access-qmrct\") pod \"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3\" (UID: \"90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3\") "
Sep 07 22:39:19.240498 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.240467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3-kube-api-access-qmrct" (OuterVolumeSpecName: "kube-api-access-qmrct") pod "90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" (UID: "90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3"). InnerVolumeSpecName "kube-api-access-qmrct". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:39:19.338902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.338863    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmrct\" (UniqueName: \"kubernetes.io/projected/90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3-kube-api-access-qmrct\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:39:19.856653 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.856618    2582 generic.go:356] "Generic (PLEG): container finished" podID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" containerID="2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba" exitCode=0
Sep 07 22:39:19.857085 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.856660    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:19.857085 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.856687    2582 scope.go:117] "RemoveContainer" containerID="2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba"
Sep 07 22:39:19.858751 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:39:19.858715    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-c45sv\" is forbidden: User \"system:node:ip-10-0-137-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-80.ec2.internal' and this object" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:19.865155 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.865135    2582 scope.go:117] "RemoveContainer" containerID="2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba"
Sep 07 22:39:19.865454 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:39:19.865434    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba\": container with ID starting with 2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba not found: ID does not exist" containerID="2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba"
Sep 07 22:39:19.865512 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:19.865467    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba"} err="failed to get container status \"2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba\": rpc error: code = NotFound desc = could not find container \"2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba\": container with ID starting with 2b61f9e88a834f331d0e6e21eba7fdb30c909bd5e2e659b30095812166003dba not found: ID does not exist"
Sep 07 22:39:19.866917 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:39:19.866897    2582 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-6c549b6d85-c45sv\" is forbidden: User \"system:node:ip-10-0-137-80.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-137-80.ec2.internal' and this object" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" pod="kuadrant-system/limitador-operator-controller-manager-6c549b6d85-c45sv"
Sep 07 22:39:20.088625 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:20.088588    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" path="/var/lib/kubelet/pods/90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3/volumes"
Sep 07 22:39:34.925302 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.925262    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"]
Sep 07 22:39:34.925806 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.925577    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" containerName="manager"
Sep 07 22:39:34.925806 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.925589    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" containerName="manager"
Sep 07 22:39:34.925806 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.925655    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="90b2a1f9-1632-4cd4-b240-d6c6ef7d0aa3" containerName="manager"
Sep 07 22:39:34.931804 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.931775    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:34.944368 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.944342    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"]
Sep 07 22:39:34.947239 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.947207    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05dc44d-ac3e-432a-93d1-67dad46fc24b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-xrq2z\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:34.947377 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:34.947348    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtjdr\" (UniqueName: \"kubernetes.io/projected/b05dc44d-ac3e-432a-93d1-67dad46fc24b-kube-api-access-vtjdr\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-xrq2z\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:35.048439 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:35.048402    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtjdr\" (UniqueName: \"kubernetes.io/projected/b05dc44d-ac3e-432a-93d1-67dad46fc24b-kube-api-access-vtjdr\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-xrq2z\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:35.048643 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:35.048504    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05dc44d-ac3e-432a-93d1-67dad46fc24b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-xrq2z\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:35.048893 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:35.048871    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05dc44d-ac3e-432a-93d1-67dad46fc24b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-xrq2z\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:35.056454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:35.056429    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtjdr\" (UniqueName: \"kubernetes.io/projected/b05dc44d-ac3e-432a-93d1-67dad46fc24b-kube-api-access-vtjdr\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-xrq2z\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:35.242103 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:35.242009    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:35.522307 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:35.522260    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"]
Sep 07 22:39:35.524232 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:39:35.524201    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb05dc44d_ac3e_432a_93d1_67dad46fc24b.slice/crio-d4dfa0b0386d3a638b975d33541ca0ae85e1766d413f0c33f1b35c61d88589f2 WatchSource:0}: Error finding container d4dfa0b0386d3a638b975d33541ca0ae85e1766d413f0c33f1b35c61d88589f2: Status 404 returned error can't find the container with id d4dfa0b0386d3a638b975d33541ca0ae85e1766d413f0c33f1b35c61d88589f2
Sep 07 22:39:35.933449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:35.933412    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z" event={"ID":"b05dc44d-ac3e-432a-93d1-67dad46fc24b","Type":"ContainerStarted","Data":"d4dfa0b0386d3a638b975d33541ca0ae85e1766d413f0c33f1b35c61d88589f2"}
Sep 07 22:39:39.958233 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:39.958200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z" event={"ID":"b05dc44d-ac3e-432a-93d1-67dad46fc24b","Type":"ContainerStarted","Data":"9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35"}
Sep 07 22:39:39.958654 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:39.958272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:39.978198 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:39.978141    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z" podStartSLOduration=2.199739513 podStartE2EDuration="5.97812311s" totalImagesPullingTime="3.778383597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:39.976747794 +0000 UTC m=+446.442904788" watchObservedRunningTime="2026-09-07 22:39:39.97812311 +0000 UTC m=+446.444280106"
Sep 07 22:39:50.916587 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.916549    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"]
Sep 07 22:39:50.919157 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.919132    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.921533 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.921511    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-xwmqh\""
Sep 07 22:39:50.930486 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.930460    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"]
Sep 07 22:39:50.965989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.965960    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 22:39:50.982178 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982132    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982219    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c570c8f-eb2a-407e-b198-45185ffbdb57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982261    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982327    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982399    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982633 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982433    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982633 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982461    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:50.982633 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:50.982577    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47slz\" (UniqueName: \"kubernetes.io/projected/2c570c8f-eb2a-407e-b198-45185ffbdb57-kube-api-access-47slz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.082969 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.082928    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47slz\" (UniqueName: \"kubernetes.io/projected/2c570c8f-eb2a-407e-b198-45185ffbdb57-kube-api-access-47slz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.082987    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083017    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c570c8f-eb2a-407e-b198-45185ffbdb57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083039    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083059    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083076    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083105    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083132    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083168 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083158    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083604 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083568    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083898 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083878    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083988 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083883    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.083988 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.083900    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c570c8f-eb2a-407e-b198-45185ffbdb57-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.084132 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.084107    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.085539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.085507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.085820 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.085802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.090410 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.090381    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c570c8f-eb2a-407e-b198-45185ffbdb57-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.090707 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.090687    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47slz\" (UniqueName: \"kubernetes.io/projected/2c570c8f-eb2a-407e-b198-45185ffbdb57-kube-api-access-47slz\") pod \"maas-default-gateway-openshift-default-78bb78ff78-k86tk\" (UID: \"2c570c8f-eb2a-407e-b198-45185ffbdb57\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.229999 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.229902    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:51.507225 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.507140    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"]
Sep 07 22:39:51.510190 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:39:51.510152    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c570c8f_eb2a_407e_b198_45185ffbdb57.slice/crio-71d9abce7e7cf78853ad63083f4334c1cbf123f5ddf8ce1fd1af0444d62ac81c WatchSource:0}: Error finding container 71d9abce7e7cf78853ad63083f4334c1cbf123f5ddf8ce1fd1af0444d62ac81c: Status 404 returned error can't find the container with id 71d9abce7e7cf78853ad63083f4334c1cbf123f5ddf8ce1fd1af0444d62ac81c
Sep 07 22:39:51.512252 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.512214    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:39:51.512361 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.512310    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:39:51.512361 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:51.512345    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:39:52.017547 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:52.017509    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" event={"ID":"2c570c8f-eb2a-407e-b198-45185ffbdb57","Type":"ContainerStarted","Data":"33702dd240fdc7c36f609be9309ab4aa258002c41d9a8d54402ef4f99e0436e0"}
Sep 07 22:39:52.018001 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:52.017568    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" event={"ID":"2c570c8f-eb2a-407e-b198-45185ffbdb57","Type":"ContainerStarted","Data":"71d9abce7e7cf78853ad63083f4334c1cbf123f5ddf8ce1fd1af0444d62ac81c"}
Sep 07 22:39:52.037695 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:52.037623    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" podStartSLOduration=2.037604492 podStartE2EDuration="2.037604492s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:39:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:39:52.035493916 +0000 UTC m=+458.501650912" watchObservedRunningTime="2026-09-07 22:39:52.037604492 +0000 UTC m=+458.503761487"
Sep 07 22:39:52.230879 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:52.230844    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:52.235795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:52.235767    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:53.021777 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:53.021743    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:39:53.022686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:39:53.022668    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:40:08.562305 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.561997    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bh89v"]
Sep 07 22:40:08.567305 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.567257    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bh89v"
Sep 07 22:40:08.569709 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.569672    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bh89v"]
Sep 07 22:40:08.569838 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.569780    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-sdmsr\""
Sep 07 22:40:08.639194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.639154    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psq26\" (UniqueName: \"kubernetes.io/projected/3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e-kube-api-access-psq26\") pod \"authorino-c548b5d7d-bh89v\" (UID: \"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e\") " pod="kuadrant-system/authorino-c548b5d7d-bh89v"
Sep 07 22:40:08.740195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.740161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psq26\" (UniqueName: \"kubernetes.io/projected/3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e-kube-api-access-psq26\") pod \"authorino-c548b5d7d-bh89v\" (UID: \"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e\") " pod="kuadrant-system/authorino-c548b5d7d-bh89v"
Sep 07 22:40:08.748234 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.748195    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psq26\" (UniqueName: \"kubernetes.io/projected/3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e-kube-api-access-psq26\") pod \"authorino-c548b5d7d-bh89v\" (UID: \"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e\") " pod="kuadrant-system/authorino-c548b5d7d-bh89v"
Sep 07 22:40:08.878435 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:08.878358    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bh89v"
Sep 07 22:40:09.144206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:09.144181    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bh89v"]
Sep 07 22:40:09.146319 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:40:09.146277    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3cc12f8f_daa7_4b10_ba8a_3ae44c50a72e.slice/crio-de1bd652c560a9a0216a489e9407fcde753ef25ca6aed10d9fa831228ba35e05 WatchSource:0}: Error finding container de1bd652c560a9a0216a489e9407fcde753ef25ca6aed10d9fa831228ba35e05: Status 404 returned error can't find the container with id de1bd652c560a9a0216a489e9407fcde753ef25ca6aed10d9fa831228ba35e05
Sep 07 22:40:10.106238 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:10.106179    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bh89v" event={"ID":"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e","Type":"ContainerStarted","Data":"de1bd652c560a9a0216a489e9407fcde753ef25ca6aed10d9fa831228ba35e05"}
Sep 07 22:40:12.117575 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:12.117536    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bh89v" event={"ID":"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e","Type":"ContainerStarted","Data":"84e6d6e111b25b929295da44b801d257e2bf5106c2639dde0a746f74df177ddb"}
Sep 07 22:40:12.131786 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:12.131735    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-c548b5d7d-bh89v" podStartSLOduration=1.772681065 podStartE2EDuration="4.131720507s" totalImagesPullingTime="2.359039442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:40:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:40:12.130011184 +0000 UTC m=+478.596168179" watchObservedRunningTime="2026-09-07 22:40:12.131720507 +0000 UTC m=+478.597877529"
Sep 07 22:40:39.014709 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.014670    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bh89v"]
Sep 07 22:40:39.015097 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.014897    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-c548b5d7d-bh89v" podUID="3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e" containerName="authorino" containerID="cri-o://84e6d6e111b25b929295da44b801d257e2bf5106c2639dde0a746f74df177ddb" gracePeriod=30
Sep 07 22:40:39.248039 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.248002    2582 generic.go:356] "Generic (PLEG): container finished" podID="3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e" containerID="84e6d6e111b25b929295da44b801d257e2bf5106c2639dde0a746f74df177ddb" exitCode=0
Sep 07 22:40:39.248226 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.248080    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bh89v" event={"ID":"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e","Type":"ContainerDied","Data":"84e6d6e111b25b929295da44b801d257e2bf5106c2639dde0a746f74df177ddb"}
Sep 07 22:40:39.273468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.273396    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bh89v"
Sep 07 22:40:39.405660 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.405618    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-psq26\" (UniqueName: \"kubernetes.io/projected/3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e-kube-api-access-psq26\") pod \"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e\" (UID: \"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e\") "
Sep 07 22:40:39.407743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.407712    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e-kube-api-access-psq26" (OuterVolumeSpecName: "kube-api-access-psq26") pod "3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e" (UID: "3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e"). InnerVolumeSpecName "kube-api-access-psq26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:40:39.506848 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:39.506811    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psq26\" (UniqueName: \"kubernetes.io/projected/3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e-kube-api-access-psq26\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:40:40.254664 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:40.254623    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-c548b5d7d-bh89v" event={"ID":"3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e","Type":"ContainerDied","Data":"de1bd652c560a9a0216a489e9407fcde753ef25ca6aed10d9fa831228ba35e05"}
Sep 07 22:40:40.254664 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:40.254659    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-c548b5d7d-bh89v"
Sep 07 22:40:40.255189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:40.254678    2582 scope.go:117] "RemoveContainer" containerID="84e6d6e111b25b929295da44b801d257e2bf5106c2639dde0a746f74df177ddb"
Sep 07 22:40:40.269493 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:40.269452    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bh89v"]
Sep 07 22:40:40.271056 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:40.271036    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-c548b5d7d-bh89v"]
Sep 07 22:40:42.089623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:40:42.089587    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e" path="/var/lib/kubelet/pods/3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e/volumes"
Sep 07 22:41:20.880187 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.880143    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"]
Sep 07 22:41:20.880748 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.880647    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e" containerName="authorino"
Sep 07 22:41:20.880748 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.880667    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e" containerName="authorino"
Sep 07 22:41:20.880868 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.880757    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc12f8f-daa7-4b10-ba8a-3ae44c50a72e" containerName="authorino"
Sep 07 22:41:20.885136 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.885110    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:20.887476 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.887445    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-vk8m9\""
Sep 07 22:41:20.887476 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.887466    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 07 22:41:20.891270 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.891242    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"]
Sep 07 22:41:20.973934 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.973890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5d9f8-0bb1-440e-9376-7fab55da0edb-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r4nkj\" (UID: \"e2d5d9f8-0bb1-440e-9376-7fab55da0edb\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:20.973934 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:20.973935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nm4gn\" (UniqueName: \"kubernetes.io/projected/e2d5d9f8-0bb1-440e-9376-7fab55da0edb-kube-api-access-nm4gn\") pod \"payload-pre-processing-59fbc47bf8-r4nkj\" (UID: \"e2d5d9f8-0bb1-440e-9376-7fab55da0edb\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:21.074768 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:21.074730    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5d9f8-0bb1-440e-9376-7fab55da0edb-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r4nkj\" (UID: \"e2d5d9f8-0bb1-440e-9376-7fab55da0edb\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:21.074949 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:21.074780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nm4gn\" (UniqueName: \"kubernetes.io/projected/e2d5d9f8-0bb1-440e-9376-7fab55da0edb-kube-api-access-nm4gn\") pod \"payload-pre-processing-59fbc47bf8-r4nkj\" (UID: \"e2d5d9f8-0bb1-440e-9376-7fab55da0edb\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:21.075517 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:21.075495    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2d5d9f8-0bb1-440e-9376-7fab55da0edb-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-r4nkj\" (UID: \"e2d5d9f8-0bb1-440e-9376-7fab55da0edb\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:21.081824 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:21.081802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nm4gn\" (UniqueName: \"kubernetes.io/projected/e2d5d9f8-0bb1-440e-9376-7fab55da0edb-kube-api-access-nm4gn\") pod \"payload-pre-processing-59fbc47bf8-r4nkj\" (UID: \"e2d5d9f8-0bb1-440e-9376-7fab55da0edb\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:21.197783 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:21.197668    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:21.475690 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:21.475662    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"]
Sep 07 22:41:21.477908 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:41:21.477878    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2d5d9f8_0bb1_440e_9376_7fab55da0edb.slice/crio-11b96273e72194061d7fa709c2e2c7d0d44329995cdd9d26e8cb31e9cd3cdb90 WatchSource:0}: Error finding container 11b96273e72194061d7fa709c2e2c7d0d44329995cdd9d26e8cb31e9cd3cdb90: Status 404 returned error can't find the container with id 11b96273e72194061d7fa709c2e2c7d0d44329995cdd9d26e8cb31e9cd3cdb90
Sep 07 22:41:22.456898 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:22.456854    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj" event={"ID":"e2d5d9f8-0bb1-440e-9376-7fab55da0edb","Type":"ContainerStarted","Data":"11b96273e72194061d7fa709c2e2c7d0d44329995cdd9d26e8cb31e9cd3cdb90"}
Sep 07 22:41:24.467995 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:24.467957    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj" event={"ID":"e2d5d9f8-0bb1-440e-9376-7fab55da0edb","Type":"ContainerStarted","Data":"be4b9c517952c03aa878a820604b0d8dda1bf9ab238133438fb7b236be8110a0"}
Sep 07 22:41:24.468388 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:24.468074    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:24.483364 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:24.483314    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj" podStartSLOduration=1.6178933130000002 podStartE2EDuration="4.483298801s" totalImagesPullingTime="2.865405488s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:41:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:41:24.480965293 +0000 UTC m=+550.947122384" watchObservedRunningTime="2026-09-07 22:41:24.483298801 +0000 UTC m=+550.949455793"
Sep 07 22:41:30.496737 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:30.496635    2582 generic.go:356] "Generic (PLEG): container finished" podID="2c570c8f-eb2a-407e-b198-45185ffbdb57" containerID="33702dd240fdc7c36f609be9309ab4aa258002c41d9a8d54402ef4f99e0436e0" exitCode=0
Sep 07 22:41:30.496737 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:30.496676    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" event={"ID":"2c570c8f-eb2a-407e-b198-45185ffbdb57","Type":"ContainerDied","Data":"33702dd240fdc7c36f609be9309ab4aa258002c41d9a8d54402ef4f99e0436e0"}
Sep 07 22:41:30.497182 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:30.497107    2582 scope.go:117] "RemoveContainer" containerID="33702dd240fdc7c36f609be9309ab4aa258002c41d9a8d54402ef4f99e0436e0"
Sep 07 22:41:30.498265 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:30.498233    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:41:30.498352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:30.498325    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:41:30.498390 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:30.498359    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:41:31.503881 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:31.503839    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" event={"ID":"2c570c8f-eb2a-407e-b198-45185ffbdb57","Type":"ContainerStarted","Data":"d5c9bc00616a95db3db36f50c808b1c477ab04cb7e5864d1caa1b106ce98e4fa"}
Sep 07 22:41:32.230193 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:32.230153    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:41:32.230461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:32.230426    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-k86tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 07 22:41:32.230544 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:32.230502    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" podUID="2c570c8f-eb2a-407e-b198-45185ffbdb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 07 22:41:33.230983 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:33.230940    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-k86tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 07 22:41:33.231623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:33.231587    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" podUID="2c570c8f-eb2a-407e-b198-45185ffbdb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 07 22:41:34.230969 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:34.230928    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-k86tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 07 22:41:34.231269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:34.231237    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" podUID="2c570c8f-eb2a-407e-b198-45185ffbdb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 07 22:41:35.230430 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:35.230393    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-k86tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 07 22:41:35.230698 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:35.230668    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" podUID="2c570c8f-eb2a-407e-b198-45185ffbdb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 07 22:41:35.474549 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:35.474522    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-r4nkj"
Sep 07 22:41:36.230454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:36.230410    2582 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-k86tk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused" start-of-body=
Sep 07 22:41:36.230801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:36.230766    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk" podUID="2c570c8f-eb2a-407e-b198-45185ffbdb57" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.30:15021/healthz/ready\": dial tcp 10.134.0.30:15021: connect: connection refused"
Sep 07 22:41:37.234737 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:37.234701    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:41:37.235178 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:37.234984    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:41:37.235864 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:37.235844    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-k86tk"
Sep 07 22:41:49.352597 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.352558    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"]
Sep 07 22:41:49.355987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.355955    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.358419 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.358395    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 07 22:41:49.359321 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.359303    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 07 22:41:49.359432 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.359333    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Sep 07 22:41:49.359498 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.359457    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-tklfr\""
Sep 07 22:41:49.364414 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.364390    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"]
Sep 07 22:41:49.533726 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.533694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.533913 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.533745    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.533913 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.533770    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgrh4\" (UniqueName: \"kubernetes.io/projected/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-kube-api-access-xgrh4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.533913 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.533853    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.533913 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.533895    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.534064 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.533935    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.634969 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.634892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.634969 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.634946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.635175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.634999    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.635175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.635030    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgrh4\" (UniqueName: \"kubernetes.io/projected/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-kube-api-access-xgrh4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.635175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.635081    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.635175 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.635148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.635429 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.635303    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.635429 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.635416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.635571 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.635550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.637377 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.637355    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.637666 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.637642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.642572 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.642546    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgrh4\" (UniqueName: \"kubernetes.io/projected/56233dc2-70d5-46f5-aeeb-390ac5bd6dd0-kube-api-access-xgrh4\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm\" (UID: \"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.667609 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.667566    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:49.940171 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:49.940143    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"]
Sep 07 22:41:49.943150 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:41:49.943111    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56233dc2_70d5_46f5_aeeb_390ac5bd6dd0.slice/crio-a59106e0e54b7807b47a35e946a619fd81c36975cdbd395be47efd4b2e42fc19 WatchSource:0}: Error finding container a59106e0e54b7807b47a35e946a619fd81c36975cdbd395be47efd4b2e42fc19: Status 404 returned error can't find the container with id a59106e0e54b7807b47a35e946a619fd81c36975cdbd395be47efd4b2e42fc19
Sep 07 22:41:50.610952 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:50.610913    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm" event={"ID":"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0","Type":"ContainerStarted","Data":"a59106e0e54b7807b47a35e946a619fd81c36975cdbd395be47efd4b2e42fc19"}
Sep 07 22:41:51.617603 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:51.617560    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm" event={"ID":"56233dc2-70d5-46f5-aeeb-390ac5bd6dd0","Type":"ContainerStarted","Data":"7d001b298e6780a3c9f73f78e05ff875e54aa8e66e88c8e94557634e9d906378"}
Sep 07 22:41:51.635006 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:51.634955    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm" podStartSLOduration=1.5263414229999999 podStartE2EDuration="2.634938221s" totalImagesPullingTime="1.108596798s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:41:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:41:51.633607724 +0000 UTC m=+578.099764723" watchObservedRunningTime="2026-09-07 22:41:51.634938221 +0000 UTC m=+578.101095216"
Sep 07 22:41:59.668120 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:59.668082    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:59.668569 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:59.668181    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:41:59.680525 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:41:59.680495    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:42:00.369937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.369903    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"]
Sep 07 22:42:00.373479 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.373454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.375802 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.375781    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 07 22:42:00.385330 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.385279    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"]
Sep 07 22:42:00.433379 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.433345    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.433582 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.433389    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.433582 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.433423    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swxsd\" (UniqueName: \"kubernetes.io/projected/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-kube-api-access-swxsd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.433582 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.433530    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.433747 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.433591    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.433747 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.433627    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.534522 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.534475    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.534522 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.534527    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.534795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.534566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.534795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.534591    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.534795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.534610    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swxsd\" (UniqueName: \"kubernetes.io/projected/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-kube-api-access-swxsd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.534795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.534647    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.535004 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.534981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.535051 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.535029    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.535087 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.535044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.537223 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.537192    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.537539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.537520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.542454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.542427    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swxsd\" (UniqueName: \"kubernetes.io/projected/f3f4c7c8-bef6-4dab-8748-c5b0cdce87de-kube-api-access-swxsd\") pod \"e2e-distinct-simulated-kserve-59975d69f9-4g5b8\" (UID: \"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.672528 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.672442    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm"
Sep 07 22:42:00.691526 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.691491    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:00.971623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:00.971598    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"]
Sep 07 22:42:00.973385 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:42:00.973353    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3f4c7c8_bef6_4dab_8748_c5b0cdce87de.slice/crio-3bcda3677f2b7ce812d6df4a5bc9643fbe6568953667808ba637aea10998af23 WatchSource:0}: Error finding container 3bcda3677f2b7ce812d6df4a5bc9643fbe6568953667808ba637aea10998af23: Status 404 returned error can't find the container with id 3bcda3677f2b7ce812d6df4a5bc9643fbe6568953667808ba637aea10998af23
Sep 07 22:42:01.667683 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:01.667639    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8" event={"ID":"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de","Type":"ContainerStarted","Data":"a23047d70c76992c162019af558eaab2784b38812e8440c328f52274abdf637c"}
Sep 07 22:42:01.667882 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:01.667696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8" event={"ID":"f3f4c7c8-bef6-4dab-8748-c5b0cdce87de","Type":"ContainerStarted","Data":"3bcda3677f2b7ce812d6df4a5bc9643fbe6568953667808ba637aea10998af23"}
Sep 07 22:42:01.685417 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:01.685366    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8" podStartSLOduration=1.505498479 podStartE2EDuration="1.685350075s" totalImagesPullingTime="179.851596ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:42:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:42:01.683429601 +0000 UTC m=+588.149586624" watchObservedRunningTime="2026-09-07 22:42:01.685350075 +0000 UTC m=+588.151507070"
Sep 07 22:42:10.692411 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:10.692372    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:10.692801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:10.692476    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:10.705278 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:10.705251    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:42:10.723393 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:42:10.723362    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-4g5b8"
Sep 07 22:43:24.467944 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.467912    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"]
Sep 07 22:43:24.471459 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.471436    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.473799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.473764    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 07 22:43:24.481581 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.481558    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"]
Sep 07 22:43:24.580172 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.580135    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.580368 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.580208    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d72cce6-e5a0-4ff1-9afc-8f934265deab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.580368 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.580244    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.580368 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.580262    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.580497 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.580371    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.580497 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.580400    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgglw\" (UniqueName: \"kubernetes.io/projected/1d72cce6-e5a0-4ff1-9afc-8f934265deab-kube-api-access-bgglw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.680906 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.680875    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681077 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.680932    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d72cce6-e5a0-4ff1-9afc-8f934265deab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681077 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.680961    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681077 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.680978    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681077 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.681006    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681077 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.681023    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgglw\" (UniqueName: \"kubernetes.io/projected/1d72cce6-e5a0-4ff1-9afc-8f934265deab-kube-api-access-bgglw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681450 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.681417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.681452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.681564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.681417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.683059 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.683041    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d72cce6-e5a0-4ff1-9afc-8f934265deab-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.683537 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.683515    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d72cce6-e5a0-4ff1-9afc-8f934265deab-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.688514 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.688485    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgglw\" (UniqueName: \"kubernetes.io/projected/1d72cce6-e5a0-4ff1-9afc-8f934265deab-kube-api-access-bgglw\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh\" (UID: \"1d72cce6-e5a0-4ff1-9afc-8f934265deab\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:24.781654 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:24.781554    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:25.054892 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:25.054863    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"]
Sep 07 22:43:25.062026 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:25.059960    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:43:25.069858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:25.069827    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh" event={"ID":"1d72cce6-e5a0-4ff1-9afc-8f934265deab","Type":"ContainerStarted","Data":"e0d71e8521f042a2c4a3a322bd3ff30af4bee1ff80871b5b6a7c7a2fcdb209c5"}
Sep 07 22:43:26.076743 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:26.076696    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh" event={"ID":"1d72cce6-e5a0-4ff1-9afc-8f934265deab","Type":"ContainerStarted","Data":"9983425d5368ebee74dc2fa9b9467d10eb0c7bd42cf574525d806dab4931ade8"}
Sep 07 22:43:26.092564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:26.092513    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh" podStartSLOduration=1.904647077 podStartE2EDuration="2.092497657s" totalImagesPullingTime="187.85058ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:43:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:43:26.091078718 +0000 UTC m=+672.557235713" watchObservedRunningTime="2026-09-07 22:43:26.092497657 +0000 UTC m=+672.558654651"
Sep 07 22:43:34.782390 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:34.782345    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:34.782931 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:34.782411    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:34.795397 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:34.795361    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:43:35.133113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:43:35.133078    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh"
Sep 07 22:45:00.135686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:00.135644    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75"]
Sep 07 22:45:00.139024 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:00.139003    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75"
Sep 07 22:45:00.141496 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:00.141467    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:45:00.142850 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:00.142818    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75"]
Sep 07 22:45:00.150334 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:00.150303    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75"
Sep 07 22:45:00.419954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:00.419923    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75"]
Sep 07 22:45:00.422113 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:45:00.422081    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef5395e7_5f04_4661_9296_fb0f13fff291.slice/crio-b560bf8b7475967212f75518016d75a3d12c994c9528d5d658e0f185c88d2988 WatchSource:0}: Error finding container b560bf8b7475967212f75518016d75a3d12c994c9528d5d658e0f185c88d2988: Status 404 returned error can't find the container with id b560bf8b7475967212f75518016d75a3d12c994c9528d5d658e0f185c88d2988
Sep 07 22:45:00.551949 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:00.551915    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75" event={"ID":"ef5395e7-5f04-4661-9296-fb0f13fff291","Type":"ContainerStarted","Data":"b560bf8b7475967212f75518016d75a3d12c994c9528d5d658e0f185c88d2988"}
Sep 07 22:45:03.568363 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:03.568321    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75" event={"ID":"ef5395e7-5f04-4661-9296-fb0f13fff291","Type":"ContainerStarted","Data":"cebe574a8faac4e96751ce0f719fa94e4ee34c5b7054e5f857155b8c6c88f111"}
Sep 07 22:45:03.580996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:03.580944    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75" podStartSLOduration=1.144049781 podStartE2EDuration="3.580928328s" totalImagesPullingTime="2.436878547s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:03.580100428 +0000 UTC m=+770.046257424" watchObservedRunningTime="2026-09-07 22:45:03.580928328 +0000 UTC m=+770.047085323"
Sep 07 22:45:04.575254 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:04.575217    2582 generic.go:356] "Generic (PLEG): container finished" podID="ef5395e7-5f04-4661-9296-fb0f13fff291" containerID="cebe574a8faac4e96751ce0f719fa94e4ee34c5b7054e5f857155b8c6c88f111" exitCode=0
Sep 07 22:45:04.575713 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:04.575255    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75" event={"ID":"ef5395e7-5f04-4661-9296-fb0f13fff291","Type":"ContainerDied","Data":"cebe574a8faac4e96751ce0f719fa94e4ee34c5b7054e5f857155b8c6c88f111"}
Sep 07 22:45:05.714934 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:05.714899    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75"
Sep 07 22:45:06.272169 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.272136    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"]
Sep 07 22:45:06.272652 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.272614    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef5395e7-5f04-4661-9296-fb0f13fff291" containerName="cleanup"
Sep 07 22:45:06.272652 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.272638    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef5395e7-5f04-4661-9296-fb0f13fff291" containerName="cleanup"
Sep 07 22:45:06.272849 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.272771    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef5395e7-5f04-4661-9296-fb0f13fff291" containerName="cleanup"
Sep 07 22:45:06.275952 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.275930    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.278460 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.278432    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-51cb36-openshift-default-dockercfg-fnsj8\""
Sep 07 22:45:06.292504 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.292477    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"]
Sep 07 22:45:06.402378 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402274    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-data\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402378 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm88x\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-kube-api-access-tm88x\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402463    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-credential-socket\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402492    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-envoy\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402527    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-socket\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402543    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e2d636-9b36-4de1-b874-17954e4ec581-istio-podinfo\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402685 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-certs\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402685 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402590    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e2d636-9b36-4de1-b874-17954e4ec581-istiod-ca-cert\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.402685 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.402636    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-istio-token\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503565 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503524    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-credential-socket\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503565 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-envoy\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503771 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503592    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-socket\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503771 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503708    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e2d636-9b36-4de1-b874-17954e4ec581-istio-podinfo\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503771 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503756    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-certs\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503918 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503780    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e2d636-9b36-4de1-b874-17954e4ec581-istiod-ca-cert\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503918 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503811    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-istio-token\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.503918 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503886    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-data\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.504067 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503920    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm88x\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-kube-api-access-tm88x\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.504067 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.503987    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-socket\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.504067 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.504047    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-credential-socket\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.504302 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.504218    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-certs\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.504302 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.504239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-data\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.504890 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.504865    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e2d636-9b36-4de1-b874-17954e4ec581-istiod-ca-cert\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.505933 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.505911    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-envoy\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.506523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.506505    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e2d636-9b36-4de1-b874-17954e4ec581-istio-podinfo\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.511438 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.511417    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-istio-token\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.511658 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.511638    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm88x\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-kube-api-access-tm88x\") pod \"e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") " pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.586096 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.586061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75" event={"ID":"ef5395e7-5f04-4661-9296-fb0f13fff291","Type":"ContainerDied","Data":"b560bf8b7475967212f75518016d75a3d12c994c9528d5d658e0f185c88d2988"}
Sep 07 22:45:06.586096 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.586082    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29813685-lzn75"
Sep 07 22:45:06.586375 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.586115    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b560bf8b7475967212f75518016d75a3d12c994c9528d5d658e0f185c88d2988"
Sep 07 22:45:06.586375 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.586261    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:06.876647 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.876613    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"]
Sep 07 22:45:06.879359 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:45:06.879326    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e2d636_9b36_4de1_b874_17954e4ec581.slice/crio-ae7f9a5bf59f04a57ce692c9adae990e424a634e74ae99843d6baabdde6d1309 WatchSource:0}: Error finding container ae7f9a5bf59f04a57ce692c9adae990e424a634e74ae99843d6baabdde6d1309: Status 404 returned error can't find the container with id ae7f9a5bf59f04a57ce692c9adae990e424a634e74ae99843d6baabdde6d1309
Sep 07 22:45:06.881696 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.881648    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:06.881777 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.881737    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:06.881777 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:06.881772    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:07.592206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:07.592173    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" event={"ID":"38e2d636-9b36-4de1-b874-17954e4ec581","Type":"ContainerStarted","Data":"36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9"}
Sep 07 22:45:07.592410 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:07.592225    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" event={"ID":"38e2d636-9b36-4de1-b874-17954e4ec581","Type":"ContainerStarted","Data":"ae7f9a5bf59f04a57ce692c9adae990e424a634e74ae99843d6baabdde6d1309"}
Sep 07 22:45:07.611174 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:07.611107    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podStartSLOduration=1.6110886930000001 podStartE2EDuration="1.611088693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:07.610671803 +0000 UTC m=+774.076828790" watchObservedRunningTime="2026-09-07 22:45:07.611088693 +0000 UTC m=+774.077245765"
Sep 07 22:45:08.587495 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:08.587445    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:08.592971 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:08.592941    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:08.597341 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:08.597311    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:08.598409 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:08.598386    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:11.491459 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.491421    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"]
Sep 07 22:45:11.494623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.494602    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.496842 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.496813    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-51cb36\""
Sep 07 22:45:11.496971 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.496861    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-51cb36-dockercfg-vx9w4\""
Sep 07 22:45:11.504706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.504678    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"]
Sep 07 22:45:11.519741 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.519703    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"]
Sep 07 22:45:11.523206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.523179    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:11.537746 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.537720    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"]
Sep 07 22:45:11.545781 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.545752    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjz8k\" (UniqueName: \"kubernetes.io/projected/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-kube-api-access-pjz8k\") pod \"payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:11.545920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.545794    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833778f4-e80f-4ef0-baa5-2159e6175668-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.545920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.545832    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:11.546000 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.545930    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x8w6p\" (UniqueName: \"kubernetes.io/projected/833778f4-e80f-4ef0-baa5-2159e6175668-kube-api-access-x8w6p\") pod \"payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.646873 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.646829    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x8w6p\" (UniqueName: \"kubernetes.io/projected/833778f4-e80f-4ef0-baa5-2159e6175668-kube-api-access-x8w6p\") pod \"payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.647061 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.646885    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjz8k\" (UniqueName: \"kubernetes.io/projected/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-kube-api-access-pjz8k\") pod \"payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:11.647061 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.646902    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833778f4-e80f-4ef0-baa5-2159e6175668-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.647061 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.646949    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:11.647708 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.647683    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:11.647801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.647716    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833778f4-e80f-4ef0-baa5-2159e6175668-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.656978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.656954    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x8w6p\" (UniqueName: \"kubernetes.io/projected/833778f4-e80f-4ef0-baa5-2159e6175668-kube-api-access-x8w6p\") pod \"payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.657083 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.657064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjz8k\" (UniqueName: \"kubernetes.io/projected/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-kube-api-access-pjz8k\") pod \"payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") " pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:11.805920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.805890    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:11.834110 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:11.834076    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:12.097157 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.097128    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"]
Sep 07 22:45:12.120825 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.120793    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"]
Sep 07 22:45:12.136546 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:45:12.136514    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf3bb18c_95db_4e97_bfcf_c95641ba8a6c.slice/crio-128e0b4117fbe8753fcbf09c29057150a5cb16fde9a10fca76c1c6b04ef20cd0 WatchSource:0}: Error finding container 128e0b4117fbe8753fcbf09c29057150a5cb16fde9a10fca76c1c6b04ef20cd0: Status 404 returned error can't find the container with id 128e0b4117fbe8753fcbf09c29057150a5cb16fde9a10fca76c1c6b04ef20cd0
Sep 07 22:45:12.625186 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.625139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh" event={"ID":"df3bb18c-95db-4e97-bfcf-c95641ba8a6c","Type":"ContainerStarted","Data":"b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87"}
Sep 07 22:45:12.625708 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.625211    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh" event={"ID":"df3bb18c-95db-4e97-bfcf-c95641ba8a6c","Type":"ContainerStarted","Data":"128e0b4117fbe8753fcbf09c29057150a5cb16fde9a10fca76c1c6b04ef20cd0"}
Sep 07 22:45:12.625708 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.625308    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:12.627295 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.627261    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8" event={"ID":"833778f4-e80f-4ef0-baa5-2159e6175668","Type":"ContainerStarted","Data":"eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc"}
Sep 07 22:45:12.627460 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.627331    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:12.627460 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.627357    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8" event={"ID":"833778f4-e80f-4ef0-baa5-2159e6175668","Type":"ContainerStarted","Data":"7cf3e419b8e9f80ceb23723acf4893ab5a90e5cb95b6ebf50168ff74260ab6b0"}
Sep 07 22:45:12.641894 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.641837    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh" podStartSLOduration=1.6418193730000001 podStartE2EDuration="1.641819373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:12.639550225 +0000 UTC m=+779.105707224" watchObservedRunningTime="2026-09-07 22:45:12.641819373 +0000 UTC m=+779.107976367"
Sep 07 22:45:12.656758 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:12.656701    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8" podStartSLOduration=1.6566838750000001 podStartE2EDuration="1.656683875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:12.655803604 +0000 UTC m=+779.121960600" watchObservedRunningTime="2026-09-07 22:45:12.656683875 +0000 UTC m=+779.122840870"
Sep 07 22:45:14.255190 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.255143    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"]
Sep 07 22:45:14.258427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.258395    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.260679 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.260654    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-autores-7c3369-openshift-default-dockercfg-66h9z\""
Sep 07 22:45:14.269128 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269099    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-data\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269261 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269146    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-certs\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269261 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269177    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-credential-socket\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269261 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269251    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-socket\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269396 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269308    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc9ch\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-kube-api-access-tc9ch\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269396 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269354    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f54d5-c723-4dc7-b565-54db8e10122c-istio-podinfo\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269396 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-envoy\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269485 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269441    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f54d5-c723-4dc7-b565-54db8e10122c-istiod-ca-cert\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269485 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269466    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-istio-token\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.269886 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.269867    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"]
Sep 07 22:45:14.370158 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-socket\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370158 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370161    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc9ch\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-kube-api-access-tc9ch\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370211    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f54d5-c723-4dc7-b565-54db8e10122c-istio-podinfo\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370247    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-envoy\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370306    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f54d5-c723-4dc7-b565-54db8e10122c-istiod-ca-cert\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370342    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-istio-token\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370368    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-data\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370405    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-certs\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.370449 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370438    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-credential-socket\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.371012 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.370986    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-credential-socket\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.371086 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.371072    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-socket\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.371353 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.371328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-data\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.371578 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.371554    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-certs\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.372331 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.372305    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f54d5-c723-4dc7-b565-54db8e10122c-istiod-ca-cert\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.374475 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.374445    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-envoy\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.374623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.374605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f54d5-c723-4dc7-b565-54db8e10122c-istio-podinfo\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.378264 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.378239    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc9ch\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-kube-api-access-tc9ch\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.378408 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.378272    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-istio-token\") pod \"e2e-autores-7c3369-openshift-default-674b75f5b-ws78j\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") " pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.569732 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.569692    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:14.855161 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.855133    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"]
Sep 07 22:45:14.857298 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:45:14.857250    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod569f54d5_c723_4dc7_b565_54db8e10122c.slice/crio-2653ea6c90910f9aee738e28bab9c2c28640c50a0616ae9f57efc943c5db48f9 WatchSource:0}: Error finding container 2653ea6c90910f9aee738e28bab9c2c28640c50a0616ae9f57efc943c5db48f9: Status 404 returned error can't find the container with id 2653ea6c90910f9aee738e28bab9c2c28640c50a0616ae9f57efc943c5db48f9
Sep 07 22:45:14.859512 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.859478    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:14.859608 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.859548    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:14.859608 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:14.859578    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:15.647093 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:15.647053    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" event={"ID":"569f54d5-c723-4dc7-b565-54db8e10122c","Type":"ContainerStarted","Data":"7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b"}
Sep 07 22:45:15.647524 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:15.647166    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" event={"ID":"569f54d5-c723-4dc7-b565-54db8e10122c","Type":"ContainerStarted","Data":"2653ea6c90910f9aee738e28bab9c2c28640c50a0616ae9f57efc943c5db48f9"}
Sep 07 22:45:15.668613 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:15.668554    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" podStartSLOduration=1.668537652 podStartE2EDuration="1.668537652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:15.667349416 +0000 UTC m=+782.133506413" watchObservedRunningTime="2026-09-07 22:45:15.668537652 +0000 UTC m=+782.134694647"
Sep 07 22:45:16.570390 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:16.570344    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:16.575397 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:16.575370    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:16.652198 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:16.652162    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:16.653746 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:16.653718    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:22.141354 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.141319    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"]
Sep 07 22:45:22.145569 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.145545    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.148264 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.148238    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-0e3b5bcb-dockercfg-x7fvd\""
Sep 07 22:45:22.148402 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.148323    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-0e3b5bcb\""
Sep 07 22:45:22.149119 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.149097    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"]
Sep 07 22:45:22.152595 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.152575    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.153858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.153690    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"]
Sep 07 22:45:22.167270 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.167240    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"]
Sep 07 22:45:22.266489 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.266432    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c182844-a490-4b9a-8bdf-8b5edacd3a04-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.266684 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.266501    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-957qq\" (UniqueName: \"kubernetes.io/projected/3c182844-a490-4b9a-8bdf-8b5edacd3a04-kube-api-access-957qq\") pod \"payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.266684 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.266654    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.266806 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.266694    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qdrcm\" (UniqueName: \"kubernetes.io/projected/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-kube-api-access-qdrcm\") pod \"payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.367981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.367938    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.367981 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.367980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qdrcm\" (UniqueName: \"kubernetes.io/projected/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-kube-api-access-qdrcm\") pod \"payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.368262 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.368026    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c182844-a490-4b9a-8bdf-8b5edacd3a04-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.368262 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.368048    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-957qq\" (UniqueName: \"kubernetes.io/projected/3c182844-a490-4b9a-8bdf-8b5edacd3a04-kube-api-access-957qq\") pod \"payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.368980 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.368953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c182844-a490-4b9a-8bdf-8b5edacd3a04-plugins-config-volume\") pod \"payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.369017 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.368953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.376614 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.376583    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qdrcm\" (UniqueName: \"kubernetes.io/projected/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-kube-api-access-qdrcm\") pod \"payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.376614 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.376607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-957qq\" (UniqueName: \"kubernetes.io/projected/3c182844-a490-4b9a-8bdf-8b5edacd3a04-kube-api-access-957qq\") pod \"payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") " pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.458027 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.457917    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:22.466459 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.466434    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:22.755078 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.755048    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"]
Sep 07 22:45:22.757088 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:45:22.757058    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33b9c1b2_eb85_4c1e_b836_5a826d6ff5d4.slice/crio-8de29ee5483383592e594667743ca0f4ff0be63bffc7a85619c9290ec50723e9 WatchSource:0}: Error finding container 8de29ee5483383592e594667743ca0f4ff0be63bffc7a85619c9290ec50723e9: Status 404 returned error can't find the container with id 8de29ee5483383592e594667743ca0f4ff0be63bffc7a85619c9290ec50723e9
Sep 07 22:45:22.786269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:22.786245    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"]
Sep 07 22:45:22.790436 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:45:22.790411    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c182844_a490_4b9a_8bdf_8b5edacd3a04.slice/crio-20abeb958adc3f945c972a7826e4841e85b29f864f056ee14c86079398a9e439 WatchSource:0}: Error finding container 20abeb958adc3f945c972a7826e4841e85b29f864f056ee14c86079398a9e439: Status 404 returned error can't find the container with id 20abeb958adc3f945c972a7826e4841e85b29f864f056ee14c86079398a9e439
Sep 07 22:45:23.637726 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.637691    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:45:23.638137 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.637809    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:45:23.692617 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.692577    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl" event={"ID":"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4","Type":"ContainerStarted","Data":"8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db"}
Sep 07 22:45:23.692799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.692643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl" event={"ID":"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4","Type":"ContainerStarted","Data":"8de29ee5483383592e594667743ca0f4ff0be63bffc7a85619c9290ec50723e9"}
Sep 07 22:45:23.692799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.692687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:23.694733 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.694691    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx" event={"ID":"3c182844-a490-4b9a-8bdf-8b5edacd3a04","Type":"ContainerStarted","Data":"01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84"}
Sep 07 22:45:23.694867 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.694751    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:23.694867 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.694778    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx" event={"ID":"3c182844-a490-4b9a-8bdf-8b5edacd3a04","Type":"ContainerStarted","Data":"20abeb958adc3f945c972a7826e4841e85b29f864f056ee14c86079398a9e439"}
Sep 07 22:45:23.710441 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.710386    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl" podStartSLOduration=1.710368639 podStartE2EDuration="1.710368639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:23.708264485 +0000 UTC m=+790.174421482" watchObservedRunningTime="2026-09-07 22:45:23.710368639 +0000 UTC m=+790.176525638"
Sep 07 22:45:23.725220 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:23.725151    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx" podStartSLOduration=1.725127635 podStartE2EDuration="1.725127635s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:23.722715332 +0000 UTC m=+790.188872325" watchObservedRunningTime="2026-09-07 22:45:23.725127635 +0000 UTC m=+790.191284631"
Sep 07 22:45:24.638505 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:24.638457    2582 patch_prober.go:28] interesting pod/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:45:24.638911 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:24.638529    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:45:27.721165 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:27.721130    2582 generic.go:356] "Generic (PLEG): container finished" podID="38e2d636-9b36-4de1-b874-17954e4ec581" containerID="36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9" exitCode=0
Sep 07 22:45:27.721623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:27.721199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" event={"ID":"38e2d636-9b36-4de1-b874-17954e4ec581","Type":"ContainerDied","Data":"36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9"}
Sep 07 22:45:27.721623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:27.721619    2582 scope.go:117] "RemoveContainer" containerID="36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9"
Sep 07 22:45:27.722715 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:27.722682    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:27.722775 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:27.722761    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:27.722817 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:27.722795    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:28.732398 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:28.732348    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" event={"ID":"38e2d636-9b36-4de1-b874-17954e4ec581","Type":"ContainerStarted","Data":"5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0"}
Sep 07 22:45:29.586652 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:29.586611    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:29.586853 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:29.586835    2582 patch_prober.go:28] interesting pod/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 22:45:29.586900 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:29.586882    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 22:45:30.587419 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:30.587386    2582 patch_prober.go:28] interesting pod/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 22:45:30.587794 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:30.587441    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 22:45:31.587537 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:31.587496    2582 patch_prober.go:28] interesting pod/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 22:45:31.587937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:31.587551    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 22:45:32.586880 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:32.586836    2582 patch_prober.go:28] interesting pod/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 22:45:32.587074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:32.586900    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 22:45:33.587416 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:33.587369    2582 patch_prober.go:28] interesting pod/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 22:45:33.587815 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:33.587429    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 22:45:34.587343 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:34.587263    2582 patch_prober.go:28] interesting pod/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Sep 07 22:45:34.587555 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:34.587367    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Sep 07 22:45:34.703161 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:34.703127    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:45:34.703362 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:34.703222    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:45:35.591279 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:35.591243    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:35.591807 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:35.591631    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:35.592392 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:35.592373    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:45:37.794407 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:37.794377    2582 generic.go:356] "Generic (PLEG): container finished" podID="569f54d5-c723-4dc7-b565-54db8e10122c" containerID="7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b" exitCode=0
Sep 07 22:45:37.794813 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:37.794439    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" event={"ID":"569f54d5-c723-4dc7-b565-54db8e10122c","Type":"ContainerDied","Data":"7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b"}
Sep 07 22:45:37.794894 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:37.794877    2582 scope.go:117] "RemoveContainer" containerID="7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b"
Sep 07 22:45:37.796091 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:37.796058    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:37.796183 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:37.796143    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:37.796183 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:37.796179    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:45:38.801408 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:38.801373    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" event={"ID":"569f54d5-c723-4dc7-b565-54db8e10122c","Type":"ContainerStarted","Data":"0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969"}
Sep 07 22:45:39.545417 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.545380    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"]
Sep 07 22:45:39.550521 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.550386    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:39.553804 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.553779    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-79b9aa-dockercfg-5jtc5\""
Sep 07 22:45:39.553985 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.553963    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-79b9aa\""
Sep 07 22:45:39.562014 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.561976    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"]
Sep 07 22:45:39.570688 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.570653    2582 patch_prober.go:28] interesting pod/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 07 22:45:39.570871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.570710    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 07 22:45:39.570929 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.570891    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:39.653996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.653938    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a20c25-28a4-4267-b7ec-a19fca4b55f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:39.653996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.653995    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jgmm\" (UniqueName: \"kubernetes.io/projected/03a20c25-28a4-4267-b7ec-a19fca4b55f2-kube-api-access-5jgmm\") pod \"payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:39.754858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.754808    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a20c25-28a4-4267-b7ec-a19fca4b55f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:39.754858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.754859    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jgmm\" (UniqueName: \"kubernetes.io/projected/03a20c25-28a4-4267-b7ec-a19fca4b55f2-kube-api-access-5jgmm\") pod \"payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:39.755635 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.755613    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a20c25-28a4-4267-b7ec-a19fca4b55f2-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:39.763004 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.762978    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jgmm\" (UniqueName: \"kubernetes.io/projected/03a20c25-28a4-4267-b7ec-a19fca4b55f2-kube-api-access-5jgmm\") pod \"payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:39.864312 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:39.864236    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:40.159504 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:40.159476    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"]
Sep 07 22:45:40.161502 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:45:40.161468    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03a20c25_28a4_4267_b7ec_a19fca4b55f2.slice/crio-d98f80f412c456765b8cb671b5ae2b0d7cd3003a5ff59151049c5c2df7b9918a WatchSource:0}: Error finding container d98f80f412c456765b8cb671b5ae2b0d7cd3003a5ff59151049c5c2df7b9918a: Status 404 returned error can't find the container with id d98f80f412c456765b8cb671b5ae2b0d7cd3003a5ff59151049c5c2df7b9918a
Sep 07 22:45:40.570221 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:40.570183    2582 patch_prober.go:28] interesting pod/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 07 22:45:40.570424 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:40.570252    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 07 22:45:40.821373 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:40.814779    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5" event={"ID":"03a20c25-28a4-4267-b7ec-a19fca4b55f2","Type":"ContainerStarted","Data":"d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564"}
Sep 07 22:45:40.821373 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:40.814847    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5" event={"ID":"03a20c25-28a4-4267-b7ec-a19fca4b55f2","Type":"ContainerStarted","Data":"d98f80f412c456765b8cb671b5ae2b0d7cd3003a5ff59151049c5c2df7b9918a"}
Sep 07 22:45:40.821373 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:40.815677    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:45:40.831010 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:40.830941    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5" podStartSLOduration=1.830915875 podStartE2EDuration="1.830915875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:45:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:45:40.829719261 +0000 UTC m=+807.295876260" watchObservedRunningTime="2026-09-07 22:45:40.830915875 +0000 UTC m=+807.297072872"
Sep 07 22:45:41.570568 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:41.570526    2582 patch_prober.go:28] interesting pod/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 07 22:45:41.571094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:41.570596    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 07 22:45:42.570279 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:42.570245    2582 patch_prober.go:28] interesting pod/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 07 22:45:42.570461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:42.570329    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 07 22:45:43.570562 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:43.570523    2582 patch_prober.go:28] interesting pod/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Sep 07 22:45:43.570963 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:43.570583    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Sep 07 22:45:44.575523 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:44.575489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:44.575920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:44.575895    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:44.576728 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:44.576710    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:45:52.835661 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:45:52.835623    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:46:02.052911 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.052871    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"]
Sep 07 22:46:02.053413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.053176    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx" podUID="3c182844-a490-4b9a-8bdf-8b5edacd3a04" containerName="payload-processing" containerID="cri-o://01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84" gracePeriod=30
Sep 07 22:46:02.060954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.060923    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"]
Sep 07 22:46:02.061313 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.061165    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl" podUID="33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" containerName="payload-pre-processing" containerID="cri-o://8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db" gracePeriod=30
Sep 07 22:46:02.319724 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.319690    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:46:02.351474 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.351439    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:46:02.355916 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.355886    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-957qq\" (UniqueName: \"kubernetes.io/projected/3c182844-a490-4b9a-8bdf-8b5edacd3a04-kube-api-access-957qq\") pod \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") "
Sep 07 22:46:02.356064 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.356044    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c182844-a490-4b9a-8bdf-8b5edacd3a04-plugins-config-volume\") pod \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\" (UID: \"3c182844-a490-4b9a-8bdf-8b5edacd3a04\") "
Sep 07 22:46:02.356541 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.356510    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c182844-a490-4b9a-8bdf-8b5edacd3a04-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3c182844-a490-4b9a-8bdf-8b5edacd3a04" (UID: "3c182844-a490-4b9a-8bdf-8b5edacd3a04"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:02.358254 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.358222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c182844-a490-4b9a-8bdf-8b5edacd3a04-kube-api-access-957qq" (OuterVolumeSpecName: "kube-api-access-957qq") pod "3c182844-a490-4b9a-8bdf-8b5edacd3a04" (UID: "3c182844-a490-4b9a-8bdf-8b5edacd3a04"). InnerVolumeSpecName "kube-api-access-957qq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:02.456957 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.456913    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-plugins-config-volume\") pod \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") "
Sep 07 22:46:02.457138 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.456998    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qdrcm\" (UniqueName: \"kubernetes.io/projected/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-kube-api-access-qdrcm\") pod \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\" (UID: \"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4\") "
Sep 07 22:46:02.457251 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.457231    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c182844-a490-4b9a-8bdf-8b5edacd3a04-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.457332 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.457256    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-957qq\" (UniqueName: \"kubernetes.io/projected/3c182844-a490-4b9a-8bdf-8b5edacd3a04-kube-api-access-957qq\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.457817 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.457784    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" (UID: "33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:02.459904 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.459874    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-kube-api-access-qdrcm" (OuterVolumeSpecName: "kube-api-access-qdrcm") pod "33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" (UID: "33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4"). InnerVolumeSpecName "kube-api-access-qdrcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:02.558504 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.558469    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.558504 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.558499    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qdrcm\" (UniqueName: \"kubernetes.io/projected/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4-kube-api-access-qdrcm\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:02.942567 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.942525    2582 generic.go:356] "Generic (PLEG): container finished" podID="33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" containerID="8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db" exitCode=0
Sep 07 22:46:02.942760 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.942572    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"
Sep 07 22:46:02.942760 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.942612    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl" event={"ID":"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4","Type":"ContainerDied","Data":"8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db"}
Sep 07 22:46:02.942760 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.942698    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl" event={"ID":"33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4","Type":"ContainerDied","Data":"8de29ee5483383592e594667743ca0f4ff0be63bffc7a85619c9290ec50723e9"}
Sep 07 22:46:02.942760 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.942725    2582 scope.go:117] "RemoveContainer" containerID="8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db"
Sep 07 22:46:02.943989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.943962    2582 generic.go:356] "Generic (PLEG): container finished" podID="3c182844-a490-4b9a-8bdf-8b5edacd3a04" containerID="01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84" exitCode=0
Sep 07 22:46:02.944106 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.944003    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"
Sep 07 22:46:02.944106 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.944029    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx" event={"ID":"3c182844-a490-4b9a-8bdf-8b5edacd3a04","Type":"ContainerDied","Data":"01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84"}
Sep 07 22:46:02.944106 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.944086    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx" event={"ID":"3c182844-a490-4b9a-8bdf-8b5edacd3a04","Type":"ContainerDied","Data":"20abeb958adc3f945c972a7826e4841e85b29f864f056ee14c86079398a9e439"}
Sep 07 22:46:02.953405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.953377    2582 scope.go:117] "RemoveContainer" containerID="8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db"
Sep 07 22:46:02.953850 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:46:02.953824    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db\": container with ID starting with 8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db not found: ID does not exist" containerID="8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db"
Sep 07 22:46:02.953948 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.953862    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db"} err="failed to get container status \"8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db\": rpc error: code = NotFound desc = could not find container \"8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db\": container with ID starting with 8ba5709f4c77828903d17817e75eefb1679f6257637d448d5eb8c3e6621c24db not found: ID does not exist"
Sep 07 22:46:02.953948 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.953888    2582 scope.go:117] "RemoveContainer" containerID="01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84"
Sep 07 22:46:02.962156 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.962135    2582 scope.go:117] "RemoveContainer" containerID="01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84"
Sep 07 22:46:02.962596 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:46:02.962578    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84\": container with ID starting with 01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84 not found: ID does not exist" containerID="01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84"
Sep 07 22:46:02.962660 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.962603    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84"} err="failed to get container status \"01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84\": rpc error: code = NotFound desc = could not find container \"01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84\": container with ID starting with 01993d1feb3acd5b95440993ca51cbb69823eb1d78a75a0b3db62ba51b385c84 not found: ID does not exist"
Sep 07 22:46:02.990194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:02.990151    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"]
Sep 07 22:46:03.012306 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:03.012258    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-0e3b5bcb-589b5d7998-hhkrl"]
Sep 07 22:46:03.035569 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:03.035530    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"]
Sep 07 22:46:03.054352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:03.054319    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-0e3b5bcb-6d584668f5-q5mnx"]
Sep 07 22:46:04.089461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:04.089424    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" path="/var/lib/kubelet/pods/33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4/volumes"
Sep 07 22:46:04.089859 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:04.089815    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c182844-a490-4b9a-8bdf-8b5edacd3a04" path="/var/lib/kubelet/pods/3c182844-a490-4b9a-8bdf-8b5edacd3a04/volumes"
Sep 07 22:46:06.687365 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.687334    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"]
Sep 07 22:46:06.688002 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.687977    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" containerName="payload-pre-processing"
Sep 07 22:46:06.688002 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.688003    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" containerName="payload-pre-processing"
Sep 07 22:46:06.688149 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.688057    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c182844-a490-4b9a-8bdf-8b5edacd3a04" containerName="payload-processing"
Sep 07 22:46:06.688149 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.688067    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c182844-a490-4b9a-8bdf-8b5edacd3a04" containerName="payload-processing"
Sep 07 22:46:06.688247 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.688158    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c182844-a490-4b9a-8bdf-8b5edacd3a04" containerName="payload-processing"
Sep 07 22:46:06.688247 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.688176    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="33b9c1b2-eb85-4c1e-b836-5a826d6ff5d4" containerName="payload-pre-processing"
Sep 07 22:46:06.691094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.691074    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.704418 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.704387    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:46:06.704920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.704896    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 22:46:06.705749 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.705722    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-a602f13b-serving-cert\""
Sep 07 22:46:06.705983 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.705965    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 22:46:06.706238 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.706217    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 22:46:06.710533 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.710504    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"]
Sep 07 22:46:06.803123 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.803085    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.803355 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.803160    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-metrics-certs\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.803355 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.803213    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcjdp\" (UniqueName: \"kubernetes.io/projected/8aa2c180-c1d2-4545-bf96-0f68c08f429e-kube-api-access-gcjdp\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.803481 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.803364    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c180-c1d2-4545-bf96-0f68c08f429e-openshift-service-ca\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.904188 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.904127    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-metrics-certs\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.904188 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.904185    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcjdp\" (UniqueName: \"kubernetes.io/projected/8aa2c180-c1d2-4545-bf96-0f68c08f429e-kube-api-access-gcjdp\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.904501 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.904244    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c180-c1d2-4545-bf96-0f68c08f429e-openshift-service-ca\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.904501 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.904278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.904501 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:46:06.904392    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-serving-cert: secret "maas-api-e2e-migrate-a602f13b-serving-cert" not found
Sep 07 22:46:06.904501 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:46:06.904451    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls podName:8aa2c180-c1d2-4545-bf96-0f68c08f429e nodeName:}" failed. No retries permitted until 2026-09-07 22:46:07.40443442 +0000 UTC m=+833.870591392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls") pod "maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf" (UID: "8aa2c180-c1d2-4545-bf96-0f68c08f429e") : secret "maas-api-e2e-migrate-a602f13b-serving-cert" not found
Sep 07 22:46:06.904996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.904970    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c180-c1d2-4545-bf96-0f68c08f429e-openshift-service-ca\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.906699 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.906666    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-metrics-certs\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:06.915533 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:06.915496    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcjdp\" (UniqueName: \"kubernetes.io/projected/8aa2c180-c1d2-4545-bf96-0f68c08f429e-kube-api-access-gcjdp\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:07.286666 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.286630    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"]
Sep 07 22:46:07.289824 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.289806    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.292229 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.292206    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-a602f13b-dockercfg-8vjrc\""
Sep 07 22:46:07.292229 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.292210    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-a602f13b\""
Sep 07 22:46:07.299538 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.299512    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"]
Sep 07 22:46:07.409881 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.409843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:07.410082 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.409898    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbbcd156-87a5-4566-8f78-85c459196712-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.410082 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.409961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b7zj\" (UniqueName: \"kubernetes.io/projected/cbbcd156-87a5-4566-8f78-85c459196712-kube-api-access-2b7zj\") pod \"payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.412594 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.412563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls\") pod \"maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:07.510769 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.510731    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b7zj\" (UniqueName: \"kubernetes.io/projected/cbbcd156-87a5-4566-8f78-85c459196712-kube-api-access-2b7zj\") pod \"payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.510939 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.510827    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbbcd156-87a5-4566-8f78-85c459196712-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.511488 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.511470    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbbcd156-87a5-4566-8f78-85c459196712-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.525835 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.525804    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b7zj\" (UniqueName: \"kubernetes.io/projected/cbbcd156-87a5-4566-8f78-85c459196712-kube-api-access-2b7zj\") pod \"payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.601116 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.601071    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.617870 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.617838    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:07.885272 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.885246    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"]
Sep 07 22:46:07.886843 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:46:07.886816    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbbcd156_87a5_4566_8f78_85c459196712.slice/crio-8d17638760ba764302f2d35e6ff195b84e8fc05fb32609f46d6030b2f6f2487b WatchSource:0}: Error finding container 8d17638760ba764302f2d35e6ff195b84e8fc05fb32609f46d6030b2f6f2487b: Status 404 returned error can't find the container with id 8d17638760ba764302f2d35e6ff195b84e8fc05fb32609f46d6030b2f6f2487b
Sep 07 22:46:07.897387 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.897358    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"]
Sep 07 22:46:07.901201 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:46:07.901175    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aa2c180_c1d2_4545_bf96_0f68c08f429e.slice/crio-fac1469941f28983222e38622eaaf240a1515733661e85e4ac726771e54caf58 WatchSource:0}: Error finding container fac1469941f28983222e38622eaaf240a1515733661e85e4ac726771e54caf58: Status 404 returned error can't find the container with id fac1469941f28983222e38622eaaf240a1515733661e85e4ac726771e54caf58
Sep 07 22:46:07.975738 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.975702    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf" event={"ID":"8aa2c180-c1d2-4545-bf96-0f68c08f429e","Type":"ContainerStarted","Data":"fac1469941f28983222e38622eaaf240a1515733661e85e4ac726771e54caf58"}
Sep 07 22:46:07.977326 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.977278    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr" event={"ID":"cbbcd156-87a5-4566-8f78-85c459196712","Type":"ContainerStarted","Data":"f3f88982ed3d72e7864210ca093f3b431dcfdb91e84b4de269285e632cfaf143"}
Sep 07 22:46:07.977458 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.977351    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr" event={"ID":"cbbcd156-87a5-4566-8f78-85c459196712","Type":"ContainerStarted","Data":"8d17638760ba764302f2d35e6ff195b84e8fc05fb32609f46d6030b2f6f2487b"}
Sep 07 22:46:07.977539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.977515    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:07.992831 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:07.992769    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr" podStartSLOduration=0.992752133 podStartE2EDuration="992.752133ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:07.991712716 +0000 UTC m=+834.457869711" watchObservedRunningTime="2026-09-07 22:46:07.992752133 +0000 UTC m=+834.458909127"
Sep 07 22:46:11.003107 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:11.003067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf" event={"ID":"8aa2c180-c1d2-4545-bf96-0f68c08f429e","Type":"ContainerStarted","Data":"0d2006db2fd65b93d8a2a8e4457950cfc9e6b1039aff73ba61f6c813d54c2b06"}
Sep 07 22:46:11.003659 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:11.003265    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:11.019558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:11.019502    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf" podStartSLOduration=2.6913139790000002 podStartE2EDuration="5.019482378s" totalImagesPullingTime="2.328168399s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:11.01819327 +0000 UTC m=+837.484350265" watchObservedRunningTime="2026-09-07 22:46:11.019482378 +0000 UTC m=+837.485639378"
Sep 07 22:46:17.015388 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:17.015350    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:18.986467 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:18.986426    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:37.887418 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:37.887372    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"]
Sep 07 22:46:37.887895 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:37.887656    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf" podUID="8aa2c180-c1d2-4545-bf96-0f68c08f429e" containerName="maas-api" containerID="cri-o://0d2006db2fd65b93d8a2a8e4457950cfc9e6b1039aff73ba61f6c813d54c2b06" gracePeriod=30
Sep 07 22:46:37.949122 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:37.949079    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"]
Sep 07 22:46:37.949382 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:37.949353    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr" podUID="cbbcd156-87a5-4566-8f78-85c459196712" containerName="payload-pre-processing" containerID="cri-o://f3f88982ed3d72e7864210ca093f3b431dcfdb91e84b4de269285e632cfaf143" gracePeriod=30
Sep 07 22:46:38.154846 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.154639    2582 generic.go:356] "Generic (PLEG): container finished" podID="8aa2c180-c1d2-4545-bf96-0f68c08f429e" containerID="0d2006db2fd65b93d8a2a8e4457950cfc9e6b1039aff73ba61f6c813d54c2b06" exitCode=0
Sep 07 22:46:38.154846 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.154685    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf" event={"ID":"8aa2c180-c1d2-4545-bf96-0f68c08f429e","Type":"ContainerDied","Data":"0d2006db2fd65b93d8a2a8e4457950cfc9e6b1039aff73ba61f6c813d54c2b06"}
Sep 07 22:46:38.156852 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.156739    2582 generic.go:356] "Generic (PLEG): container finished" podID="cbbcd156-87a5-4566-8f78-85c459196712" containerID="f3f88982ed3d72e7864210ca093f3b431dcfdb91e84b4de269285e632cfaf143" exitCode=0
Sep 07 22:46:38.156852 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.156807    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr" event={"ID":"cbbcd156-87a5-4566-8f78-85c459196712","Type":"ContainerDied","Data":"f3f88982ed3d72e7864210ca093f3b431dcfdb91e84b4de269285e632cfaf143"}
Sep 07 22:46:38.215886 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.215590    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:38.249457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.249428    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:38.292902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.292860    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gcjdp\" (UniqueName: \"kubernetes.io/projected/8aa2c180-c1d2-4545-bf96-0f68c08f429e-kube-api-access-gcjdp\") pod \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") "
Sep 07 22:46:38.292902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.292907    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c180-c1d2-4545-bf96-0f68c08f429e-openshift-service-ca\") pod \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") "
Sep 07 22:46:38.293122 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.292938    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2b7zj\" (UniqueName: \"kubernetes.io/projected/cbbcd156-87a5-4566-8f78-85c459196712-kube-api-access-2b7zj\") pod \"cbbcd156-87a5-4566-8f78-85c459196712\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") "
Sep 07 22:46:38.293122 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.293001    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls\") pod \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") "
Sep 07 22:46:38.293122 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.293039    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbbcd156-87a5-4566-8f78-85c459196712-plugins-config-volume\") pod \"cbbcd156-87a5-4566-8f78-85c459196712\" (UID: \"cbbcd156-87a5-4566-8f78-85c459196712\") "
Sep 07 22:46:38.293122 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.293067    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-metrics-certs\") pod \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\" (UID: \"8aa2c180-c1d2-4545-bf96-0f68c08f429e\") "
Sep 07 22:46:38.293594 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.293396    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aa2c180-c1d2-4545-bf96-0f68c08f429e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "8aa2c180-c1d2-4545-bf96-0f68c08f429e" (UID: "8aa2c180-c1d2-4545-bf96-0f68c08f429e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:38.293594 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.293553    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbbcd156-87a5-4566-8f78-85c459196712-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbbcd156-87a5-4566-8f78-85c459196712" (UID: "cbbcd156-87a5-4566-8f78-85c459196712"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:46:38.295435 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.295399    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbbcd156-87a5-4566-8f78-85c459196712-kube-api-access-2b7zj" (OuterVolumeSpecName: "kube-api-access-2b7zj") pod "cbbcd156-87a5-4566-8f78-85c459196712" (UID: "cbbcd156-87a5-4566-8f78-85c459196712"). InnerVolumeSpecName "kube-api-access-2b7zj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:38.295569 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.295496    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aa2c180-c1d2-4545-bf96-0f68c08f429e-kube-api-access-gcjdp" (OuterVolumeSpecName: "kube-api-access-gcjdp") pod "8aa2c180-c1d2-4545-bf96-0f68c08f429e" (UID: "8aa2c180-c1d2-4545-bf96-0f68c08f429e"). InnerVolumeSpecName "kube-api-access-gcjdp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:46:38.295795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.295768    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8aa2c180-c1d2-4545-bf96-0f68c08f429e" (UID: "8aa2c180-c1d2-4545-bf96-0f68c08f429e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:46:38.295891 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.295819    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "8aa2c180-c1d2-4545-bf96-0f68c08f429e" (UID: "8aa2c180-c1d2-4545-bf96-0f68c08f429e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:46:38.394296 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.394256    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcjdp\" (UniqueName: \"kubernetes.io/projected/8aa2c180-c1d2-4545-bf96-0f68c08f429e-kube-api-access-gcjdp\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.394473 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.394324    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8aa2c180-c1d2-4545-bf96-0f68c08f429e-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.394473 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.394335    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b7zj\" (UniqueName: \"kubernetes.io/projected/cbbcd156-87a5-4566-8f78-85c459196712-kube-api-access-2b7zj\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.394473 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.394344    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.394473 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.394356    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbbcd156-87a5-4566-8f78-85c459196712-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.394473 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.394364    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8aa2c180-c1d2-4545-bf96-0f68c08f429e-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:46:38.918987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.918947    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"]
Sep 07 22:46:38.919464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.919341    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbbcd156-87a5-4566-8f78-85c459196712" containerName="payload-pre-processing"
Sep 07 22:46:38.919464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.919355    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbbcd156-87a5-4566-8f78-85c459196712" containerName="payload-pre-processing"
Sep 07 22:46:38.919464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.919366    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8aa2c180-c1d2-4545-bf96-0f68c08f429e" containerName="maas-api"
Sep 07 22:46:38.919464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.919373    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="8aa2c180-c1d2-4545-bf96-0f68c08f429e" containerName="maas-api"
Sep 07 22:46:38.919464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.919447    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="8aa2c180-c1d2-4545-bf96-0f68c08f429e" containerName="maas-api"
Sep 07 22:46:38.919464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.919458    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbbcd156-87a5-4566-8f78-85c459196712" containerName="payload-pre-processing"
Sep 07 22:46:38.922877 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.922853    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:38.925366 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.925343    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-66774a4a-openshift-default-dockercfg-hb852\""
Sep 07 22:46:38.933580 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:38.933553    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"]
Sep 07 22:46:39.000707 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000660    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-podinfo\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.000707 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000704    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-data\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.000915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000729    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-envoy\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.000915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000778    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfd15bc-8865-4659-bbcc-f95beac7c810-istiod-ca-cert\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.000915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000804    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-certs\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.000915 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000887    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-socket\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.001041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000964    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-token\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.001041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.000988    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfd7m\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-kube-api-access-zfd7m\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.001041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.001009    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-credential-socket\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102363 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102326    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-socket\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102419    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-token\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102443    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfd7m\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-kube-api-access-zfd7m\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-credential-socket\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102506    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-podinfo\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102530    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-data\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102564    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-envoy\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfd15bc-8865-4659-bbcc-f95beac7c810-istiod-ca-cert\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102623    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-certs\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.102937 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-socket\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.103100 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.102978    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-certs\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.103860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.103421    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-credential-socket\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.103860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.103829    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-data\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.103860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.103832    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfd15bc-8865-4659-bbcc-f95beac7c810-istiod-ca-cert\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.105508 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.105480    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-envoy\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.105613 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.105534    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-podinfo\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.110877 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.110591    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-token\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.110877 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.110817    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfd7m\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-kube-api-access-zfd7m\") pod \"e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") " pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.163708 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.163614    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"
Sep 07 22:46:39.163708 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.163650    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr" event={"ID":"cbbcd156-87a5-4566-8f78-85c459196712","Type":"ContainerDied","Data":"8d17638760ba764302f2d35e6ff195b84e8fc05fb32609f46d6030b2f6f2487b"}
Sep 07 22:46:39.163943 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.163721    2582 scope.go:117] "RemoveContainer" containerID="f3f88982ed3d72e7864210ca093f3b431dcfdb91e84b4de269285e632cfaf143"
Sep 07 22:46:39.165130 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.165113    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf" event={"ID":"8aa2c180-c1d2-4545-bf96-0f68c08f429e","Type":"ContainerDied","Data":"fac1469941f28983222e38622eaaf240a1515733661e85e4ac726771e54caf58"}
Sep 07 22:46:39.165192 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.165130    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"
Sep 07 22:46:39.175917 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.175859    2582 scope.go:117] "RemoveContainer" containerID="0d2006db2fd65b93d8a2a8e4457950cfc9e6b1039aff73ba61f6c813d54c2b06"
Sep 07 22:46:39.193033 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.193008    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"]
Sep 07 22:46:39.198595 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.198567    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-a602f13b-77956dd94d-6h8bf"]
Sep 07 22:46:39.208041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.208010    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"]
Sep 07 22:46:39.212095 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.212060    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-a602f13b-7ccdbf869d-svsfr"]
Sep 07 22:46:39.234803 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.234767    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:39.513587 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.513563    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"]
Sep 07 22:46:39.515717 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:46:39.515688    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cfd15bc_8865_4659_bbcc_f95beac7c810.slice/crio-aa207733d9476607a769a26dae59c2c02f5d9df8ad9fca073b991b9288a9a173 WatchSource:0}: Error finding container aa207733d9476607a769a26dae59c2c02f5d9df8ad9fca073b991b9288a9a173: Status 404 returned error can't find the container with id aa207733d9476607a769a26dae59c2c02f5d9df8ad9fca073b991b9288a9a173
Sep 07 22:46:39.518087 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.518055    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:46:39.518191 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.518122    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:46:39.518191 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:39.518156    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:46:40.090613 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:40.090571    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8aa2c180-c1d2-4545-bf96-0f68c08f429e" path="/var/lib/kubelet/pods/8aa2c180-c1d2-4545-bf96-0f68c08f429e/volumes"
Sep 07 22:46:40.091110 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:40.091008    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbbcd156-87a5-4566-8f78-85c459196712" path="/var/lib/kubelet/pods/cbbcd156-87a5-4566-8f78-85c459196712/volumes"
Sep 07 22:46:40.172730 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:40.172687    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" event={"ID":"7cfd15bc-8865-4659-bbcc-f95beac7c810","Type":"ContainerStarted","Data":"d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7"}
Sep 07 22:46:40.172927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:40.172746    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" event={"ID":"7cfd15bc-8865-4659-bbcc-f95beac7c810","Type":"ContainerStarted","Data":"aa207733d9476607a769a26dae59c2c02f5d9df8ad9fca073b991b9288a9a173"}
Sep 07 22:46:40.194549 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:40.194469    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" podStartSLOduration=2.194444382 podStartE2EDuration="2.194444382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:40.193152188 +0000 UTC m=+866.659309182" watchObservedRunningTime="2026-09-07 22:46:40.194444382 +0000 UTC m=+866.660601412"
Sep 07 22:46:40.234986 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:40.234937    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:40.240325 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:40.240292    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:41.179182 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:41.179152    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:41.180247 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:41.180227    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:46:42.630552 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.630511    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"]
Sep 07 22:46:42.643613 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.643574    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.646140 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.646107    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"]
Sep 07 22:46:42.646600 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.646398    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-66774a4a-serving-cert\""
Sep 07 22:46:42.647810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.647596    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 22:46:42.647810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.647644    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 22:46:42.647983 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.647868    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:46:42.647983 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.647899    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 22:46:42.740886 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.740849    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.741073 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.740951    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l65ms\" (UniqueName: \"kubernetes.io/projected/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-kube-api-access-l65ms\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.741073 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.741031    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-metrics-certs\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.741073 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.741057    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-openshift-service-ca\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.842396 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.842359    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-metrics-certs\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.842396 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.842401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-openshift-service-ca\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.842663 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.842463    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.842663 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.842528    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l65ms\" (UniqueName: \"kubernetes.io/projected/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-kube-api-access-l65ms\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.842663 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:46:42.842610    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-serving-cert: secret "maas-api-e2e-ait-66774a4a-serving-cert" not found
Sep 07 22:46:42.842840 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:46:42.842691    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls podName:0a10bb2d-635b-4e99-9de7-13eb43a6e8c6 nodeName:}" failed. No retries permitted until 2026-09-07 22:46:43.34266737 +0000 UTC m=+869.808824365 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls") pod "maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr" (UID: "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6") : secret "maas-api-e2e-ait-66774a4a-serving-cert" not found
Sep 07 22:46:42.843256 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.843229    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-openshift-service-ca\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.845041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.845010    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-metrics-certs\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:42.852581 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:42.852555    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l65ms\" (UniqueName: \"kubernetes.io/projected/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-kube-api-access-l65ms\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:43.347815 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:43.347774    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:43.350441 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:43.350411    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls\") pod \"maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:43.557810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:43.557763    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:43.840620 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:43.840588    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"]
Sep 07 22:46:43.842650 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:46:43.842624    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a10bb2d_635b_4e99_9de7_13eb43a6e8c6.slice/crio-bfb7b7370738b528f39f347aa8d1824b68e57baec7911617c085631d24b0852c WatchSource:0}: Error finding container bfb7b7370738b528f39f347aa8d1824b68e57baec7911617c085631d24b0852c: Status 404 returned error can't find the container with id bfb7b7370738b528f39f347aa8d1824b68e57baec7911617c085631d24b0852c
Sep 07 22:46:44.196369 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:44.196329    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr" event={"ID":"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6","Type":"ContainerStarted","Data":"bfb7b7370738b528f39f347aa8d1824b68e57baec7911617c085631d24b0852c"}
Sep 07 22:46:45.203475 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:45.203434    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr" event={"ID":"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6","Type":"ContainerStarted","Data":"d0443088286b4d6736c8168b4dfda646be7d5bc123cdfbd7c7376d6cb1c33eb2"}
Sep 07 22:46:45.203898 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:45.203576    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:46:45.222094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:45.222039    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr" podStartSLOduration=2.769889075 podStartE2EDuration="3.222021191s" totalImagesPullingTime="452.132116ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:46:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:46:45.219127705 +0000 UTC m=+871.685284697" watchObservedRunningTime="2026-09-07 22:46:45.222021191 +0000 UTC m=+871.688178186"
Sep 07 22:46:51.242548 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:46:51.242511    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:47:00.299219 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:00.299180    2582 generic.go:356] "Generic (PLEG): container finished" podID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerID="d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7" exitCode=0
Sep 07 22:47:00.299671 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:00.299251    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" event={"ID":"7cfd15bc-8865-4659-bbcc-f95beac7c810","Type":"ContainerDied","Data":"d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7"}
Sep 07 22:47:00.299715 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:00.299698    2582 scope.go:117] "RemoveContainer" containerID="d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7"
Sep 07 22:47:00.300801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:00.300774    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:47:00.300854 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:00.300841    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:47:00.300890 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:00.300876    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:47:01.306925 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:01.306888    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" event={"ID":"7cfd15bc-8865-4659-bbcc-f95beac7c810","Type":"ContainerStarted","Data":"bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf"}
Sep 07 22:47:02.234927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:02.234876    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:47:02.235436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:02.235397    2582 patch_prober.go:28] interesting pod/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 07 22:47:02.235561 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:02.235463    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 07 22:47:03.235239 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:03.235186    2582 patch_prober.go:28] interesting pod/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 07 22:47:03.235762 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:03.235264    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 07 22:47:04.236248 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:04.235772    2582 patch_prober.go:28] interesting pod/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 07 22:47:04.236248 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:04.235841    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 07 22:47:05.235653 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:05.235608    2582 patch_prober.go:28] interesting pod/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 07 22:47:05.235989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:05.235956    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 07 22:47:06.235621 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:06.235526    2582 patch_prober.go:28] interesting pod/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body=
Sep 07 22:47:06.236205 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:06.236166    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused"
Sep 07 22:47:07.240791 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:07.240755    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:47:07.241232 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:07.241170    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:47:07.242093 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:07.242069    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:47:13.190660 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.190620    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"]
Sep 07 22:47:13.191338 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.190896    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr" podUID="0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" containerName="maas-api" containerID="cri-o://d0443088286b4d6736c8168b4dfda646be7d5bc123cdfbd7c7376d6cb1c33eb2" gracePeriod=30
Sep 07 22:47:13.393750 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.393714    2582 generic.go:356] "Generic (PLEG): container finished" podID="0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" containerID="d0443088286b4d6736c8168b4dfda646be7d5bc123cdfbd7c7376d6cb1c33eb2" exitCode=0
Sep 07 22:47:13.393903 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.393768    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr" event={"ID":"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6","Type":"ContainerDied","Data":"d0443088286b4d6736c8168b4dfda646be7d5bc123cdfbd7c7376d6cb1c33eb2"}
Sep 07 22:47:13.451612 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.451541    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:47:13.555601 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.555556    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-metrics-certs\") pod \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") "
Sep 07 22:47:13.555601 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.555603    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l65ms\" (UniqueName: \"kubernetes.io/projected/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-kube-api-access-l65ms\") pod \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") "
Sep 07 22:47:13.555803 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.555746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls\") pod \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") "
Sep 07 22:47:13.555855 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.555841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-openshift-service-ca\") pod \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\" (UID: \"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6\") "
Sep 07 22:47:13.556266 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.556231    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" (UID: "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:13.557766 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.557742    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" (UID: "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:47:13.558195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.558172    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" (UID: "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:47:13.558195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.558180    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-kube-api-access-l65ms" (OuterVolumeSpecName: "kube-api-access-l65ms") pod "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" (UID: "0a10bb2d-635b-4e99-9de7-13eb43a6e8c6"). InnerVolumeSpecName "kube-api-access-l65ms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:13.656991 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.656945    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:13.656991 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.656981    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:13.656991 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.656995    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:13.657246 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:13.657006    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l65ms\" (UniqueName: \"kubernetes.io/projected/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6-kube-api-access-l65ms\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:14.400512 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:14.400476    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr" event={"ID":"0a10bb2d-635b-4e99-9de7-13eb43a6e8c6","Type":"ContainerDied","Data":"bfb7b7370738b528f39f347aa8d1824b68e57baec7911617c085631d24b0852c"}
Sep 07 22:47:14.400993 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:14.400546    2582 scope.go:117] "RemoveContainer" containerID="d0443088286b4d6736c8168b4dfda646be7d5bc123cdfbd7c7376d6cb1c33eb2"
Sep 07 22:47:14.400993 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:14.400489    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"
Sep 07 22:47:14.418955 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:14.418923    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"]
Sep 07 22:47:14.425200 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:14.425174    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-66774a4a-58cc7ccc56-j2glr"]
Sep 07 22:47:14.462572 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:14.462533    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"]
Sep 07 22:47:14.462819 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:14.462772    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy" containerID="cri-o://bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf" gracePeriod=30
Sep 07 22:47:15.323847 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.323805    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"]
Sep 07 22:47:15.324491 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.324467    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" containerName="maas-api"
Sep 07 22:47:15.324491 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.324494    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" containerName="maas-api"
Sep 07 22:47:15.324677 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.324590    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" containerName="maas-api"
Sep 07 22:47:15.329670 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.329637    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"
Sep 07 22:47:15.333179 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.333153    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:47:15.333646 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.333624    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:47:15.336028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.336001    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"]
Sep 07 22:47:15.480932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.480890    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee87d33-75cb-4a3c-b800-1e9f86e9fc4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7c3369-75pjv\" (UID: \"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"
Sep 07 22:47:15.582565 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.582467    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee87d33-75cb-4a3c-b800-1e9f86e9fc4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7c3369-75pjv\" (UID: \"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"
Sep 07 22:47:15.583154 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.583130    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee87d33-75cb-4a3c-b800-1e9f86e9fc4d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-7c3369-75pjv\" (UID: \"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"
Sep 07 22:47:15.645884 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.645834    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"
Sep 07 22:47:15.925217 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:15.925191    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"]
Sep 07 22:47:15.927795 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:47:15.927763    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbee87d33_75cb_4a3c_b800_1e9f86e9fc4d.slice/crio-d65ea5fbb95941c4a030b59b5e239bad29b3c0d5f716f10a36f5b6da4cc29cf8 WatchSource:0}: Error finding container d65ea5fbb95941c4a030b59b5e239bad29b3c0d5f716f10a36f5b6da4cc29cf8: Status 404 returned error can't find the container with id d65ea5fbb95941c4a030b59b5e239bad29b3c0d5f716f10a36f5b6da4cc29cf8
Sep 07 22:47:16.090909 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:16.090871    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a10bb2d-635b-4e99-9de7-13eb43a6e8c6" path="/var/lib/kubelet/pods/0a10bb2d-635b-4e99-9de7-13eb43a6e8c6/volumes"
Sep 07 22:47:16.414978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:16.414935    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv" event={"ID":"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d","Type":"ContainerStarted","Data":"1f8a83dfc5dcbea306fe0b28527e3effbbcacc0a379c471986d96619539a21c5"}
Sep 07 22:47:16.415176 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:16.415002    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv" event={"ID":"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d","Type":"ContainerStarted","Data":"d65ea5fbb95941c4a030b59b5e239bad29b3c0d5f716f10a36f5b6da4cc29cf8"}
Sep 07 22:47:16.430065 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:16.430007    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv" podStartSLOduration=1.429990379 podStartE2EDuration="1.429990379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:16.428298645 +0000 UTC m=+902.894455642" watchObservedRunningTime="2026-09-07 22:47:16.429990379 +0000 UTC m=+902.896147424"
Sep 07 22:47:17.422804 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:17.422766    2582 generic.go:356] "Generic (PLEG): container finished" podID="bee87d33-75cb-4a3c-b800-1e9f86e9fc4d" containerID="1f8a83dfc5dcbea306fe0b28527e3effbbcacc0a379c471986d96619539a21c5" exitCode=0
Sep 07 22:47:17.423200 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:17.422850    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv" event={"ID":"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d","Type":"ContainerDied","Data":"1f8a83dfc5dcbea306fe0b28527e3effbbcacc0a379c471986d96619539a21c5"}
Sep 07 22:47:18.566541 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.566389    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"
Sep 07 22:47:18.710905 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.710805    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee87d33-75cb-4a3c-b800-1e9f86e9fc4d-maas-api-service-ca\") pod \"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d\" (UID: \"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d\") "
Sep 07 22:47:18.711214 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.711183    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bee87d33-75cb-4a3c-b800-1e9f86e9fc4d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bee87d33-75cb-4a3c-b800-1e9f86e9fc4d" (UID: "bee87d33-75cb-4a3c-b800-1e9f86e9fc4d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:18.811825 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.811781    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bee87d33-75cb-4a3c-b800-1e9f86e9fc4d-maas-api-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:18.997707 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.997611    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"]
Sep 07 22:47:18.998271 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.998250    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bee87d33-75cb-4a3c-b800-1e9f86e9fc4d" containerName="revoke-keys"
Sep 07 22:47:18.998397 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.998276    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bee87d33-75cb-4a3c-b800-1e9f86e9fc4d" containerName="revoke-keys"
Sep 07 22:47:18.998436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:18.998398    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bee87d33-75cb-4a3c-b800-1e9f86e9fc4d" containerName="revoke-keys"
Sep 07 22:47:19.001879 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.001856    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.004335 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.004302    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-01ae3268-serving-cert\""
Sep 07 22:47:19.005257 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.005240    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 22:47:19.005416 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.005244    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 22:47:19.005416 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.005244    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 22:47:19.016183 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.010672    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"]
Sep 07 22:47:19.116537 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.116493    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-metrics-certs\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.116731 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.116547    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bnv9\" (UniqueName: \"kubernetes.io/projected/57df0612-bde2-428a-95bb-5d92e92213a0-kube-api-access-2bnv9\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.116815 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.116792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.116879 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.116829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57df0612-bde2-428a-95bb-5d92e92213a0-openshift-service-ca\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.217814 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.217768    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bnv9\" (UniqueName: \"kubernetes.io/projected/57df0612-bde2-428a-95bb-5d92e92213a0-kube-api-access-2bnv9\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.218022 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.217899    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.218022 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.217935    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57df0612-bde2-428a-95bb-5d92e92213a0-openshift-service-ca\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.218022 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.217997    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-metrics-certs\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.218197 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:47:19.218037    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-serving-cert: secret "maas-api-e2e-derive-01ae3268-serving-cert" not found
Sep 07 22:47:19.218197 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:47:19.218125    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls podName:57df0612-bde2-428a-95bb-5d92e92213a0 nodeName:}" failed. No retries permitted until 2026-09-07 22:47:19.718102003 +0000 UTC m=+906.184258983 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls") pod "maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2" (UID: "57df0612-bde2-428a-95bb-5d92e92213a0") : secret "maas-api-e2e-derive-01ae3268-serving-cert" not found
Sep 07 22:47:19.218680 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.218655    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57df0612-bde2-428a-95bb-5d92e92213a0-openshift-service-ca\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.220640 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.220618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-metrics-certs\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.231682 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.231652    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bnv9\" (UniqueName: \"kubernetes.io/projected/57df0612-bde2-428a-95bb-5d92e92213a0-kube-api-access-2bnv9\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.441153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.441124    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"
Sep 07 22:47:19.441153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.441142    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv" event={"ID":"bee87d33-75cb-4a3c-b800-1e9f86e9fc4d","Type":"ContainerDied","Data":"d65ea5fbb95941c4a030b59b5e239bad29b3c0d5f716f10a36f5b6da4cc29cf8"}
Sep 07 22:47:19.441399 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.441202    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d65ea5fbb95941c4a030b59b5e239bad29b3c0d5f716f10a36f5b6da4cc29cf8"
Sep 07 22:47:19.593434 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.593401    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"]
Sep 07 22:47:19.597004 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.596981    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.599694 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.599670    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-01ae3268-dockercfg-xtmpc\""
Sep 07 22:47:19.599803 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.599750    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-01ae3268\""
Sep 07 22:47:19.609655 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.609629    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"]
Sep 07 22:47:19.725429 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.725406    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.725503 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.725453    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b21894-98ab-4c9a-82dd-1916d498c834-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.725546 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.725522    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bgkcb\" (UniqueName: \"kubernetes.io/projected/e1b21894-98ab-4c9a-82dd-1916d498c834-kube-api-access-bgkcb\") pod \"payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.728128 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.728102    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls\") pod \"maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.749262 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.749235    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:47:19.826626 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826588    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-certs\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.826626 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826635    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-podinfo\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.826858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826689    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-token\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.826858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826704    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-socket\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.826858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826721    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfd7m\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-kube-api-access-zfd7m\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.826858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826745    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-envoy\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.826858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826835    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfd15bc-8865-4659-bbcc-f95beac7c810-istiod-ca-cert\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.827115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826890    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-credential-socket\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.827115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826923    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-data\") pod \"7cfd15bc-8865-4659-bbcc-f95beac7c810\" (UID: \"7cfd15bc-8865-4659-bbcc-f95beac7c810\") "
Sep 07 22:47:19.827115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.826934    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:19.827115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827078    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:19.827415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827166    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-data" (OuterVolumeSpecName: "istio-data") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:19.827415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827259    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:19.827415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827316    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b21894-98ab-4c9a-82dd-1916d498c834-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.827415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827401    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bgkcb\" (UniqueName: \"kubernetes.io/projected/e1b21894-98ab-4c9a-82dd-1916d498c834-kube-api-access-bgkcb\") pod \"payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.827616 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827474    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7cfd15bc-8865-4659-bbcc-f95beac7c810-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:19.827616 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827523    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.827616 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827541    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-workload-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.827616 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827554    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-credential-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.827616 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.827568    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-data\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.828161 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.828137    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b21894-98ab-4c9a-82dd-1916d498c834-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.829129 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.829098    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:47:19.829273 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.829255    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-token" (OuterVolumeSpecName: "istio-token") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:19.829447 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.829433    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:47:19.829507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.829491    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-kube-api-access-zfd7m" (OuterVolumeSpecName: "kube-api-access-zfd7m") pod "7cfd15bc-8865-4659-bbcc-f95beac7c810" (UID: "7cfd15bc-8865-4659-bbcc-f95beac7c810"). InnerVolumeSpecName "kube-api-access-zfd7m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:19.837025 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.837003    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bgkcb\" (UniqueName: \"kubernetes.io/projected/e1b21894-98ab-4c9a-82dd-1916d498c834-kube-api-access-bgkcb\") pod \"payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.908581 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.908528    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:19.919899 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.919869    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:19.928413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.928386    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-podinfo\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.928413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.928412    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-token\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.928560 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.928421    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfd7m\" (UniqueName: \"kubernetes.io/projected/7cfd15bc-8865-4659-bbcc-f95beac7c810-kube-api-access-zfd7m\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.928560 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.928430    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7cfd15bc-8865-4659-bbcc-f95beac7c810-istio-envoy\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:19.928560 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:19.928439    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7cfd15bc-8865-4659-bbcc-f95beac7c810-istiod-ca-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:20.190127 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.190101    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"]
Sep 07 22:47:20.191729 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:47:20.191696    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1b21894_98ab_4c9a_82dd_1916d498c834.slice/crio-5187f1ffe2b209752dfda1a7eab8112bd6f1e5c680fc8f369daea5a42ed106f8 WatchSource:0}: Error finding container 5187f1ffe2b209752dfda1a7eab8112bd6f1e5c680fc8f369daea5a42ed106f8: Status 404 returned error can't find the container with id 5187f1ffe2b209752dfda1a7eab8112bd6f1e5c680fc8f369daea5a42ed106f8
Sep 07 22:47:20.202094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.202065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"]
Sep 07 22:47:20.206274 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:47:20.206247    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57df0612_bde2_428a_95bb_5d92e92213a0.slice/crio-fe1c3b122672a6b565e44e07e0d2f3766c3c9dfd30fb6493483e2c646d34d78b WatchSource:0}: Error finding container fe1c3b122672a6b565e44e07e0d2f3766c3c9dfd30fb6493483e2c646d34d78b: Status 404 returned error can't find the container with id fe1c3b122672a6b565e44e07e0d2f3766c3c9dfd30fb6493483e2c646d34d78b
Sep 07 22:47:20.449235 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.449190    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw" event={"ID":"e1b21894-98ab-4c9a-82dd-1916d498c834","Type":"ContainerStarted","Data":"d6cef2a97ea97ab47d9e22c6dd54354a7d0ff5a4d9a374b9cb4dad84501d8a4a"}
Sep 07 22:47:20.449444 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.449386    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:20.449444 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.449432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw" event={"ID":"e1b21894-98ab-4c9a-82dd-1916d498c834","Type":"ContainerStarted","Data":"5187f1ffe2b209752dfda1a7eab8112bd6f1e5c680fc8f369daea5a42ed106f8"}
Sep 07 22:47:20.450488 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.450465    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2" event={"ID":"57df0612-bde2-428a-95bb-5d92e92213a0","Type":"ContainerStarted","Data":"fe1c3b122672a6b565e44e07e0d2f3766c3c9dfd30fb6493483e2c646d34d78b"}
Sep 07 22:47:20.452079 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.452053    2582 generic.go:356] "Generic (PLEG): container finished" podID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerID="bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf" exitCode=0
Sep 07 22:47:20.452192 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.452106    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"
Sep 07 22:47:20.452192 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.452112    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" event={"ID":"7cfd15bc-8865-4659-bbcc-f95beac7c810","Type":"ContainerDied","Data":"bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf"}
Sep 07 22:47:20.452331 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.452223    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd" event={"ID":"7cfd15bc-8865-4659-bbcc-f95beac7c810","Type":"ContainerDied","Data":"aa207733d9476607a769a26dae59c2c02f5d9df8ad9fca073b991b9288a9a173"}
Sep 07 22:47:20.452331 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.452257    2582 scope.go:117] "RemoveContainer" containerID="bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf"
Sep 07 22:47:20.461812 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.461787    2582 scope.go:117] "RemoveContainer" containerID="d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7"
Sep 07 22:47:20.467130 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.467080    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw" podStartSLOduration=1.467064263 podStartE2EDuration="1.467064263s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:20.465463561 +0000 UTC m=+906.931620582" watchObservedRunningTime="2026-09-07 22:47:20.467064263 +0000 UTC m=+906.933221248"
Sep 07 22:47:20.470617 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.470597    2582 scope.go:117] "RemoveContainer" containerID="bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf"
Sep 07 22:47:20.470893 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:47:20.470874    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf\": container with ID starting with bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf not found: ID does not exist" containerID="bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf"
Sep 07 22:47:20.470950 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.470904    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf"} err="failed to get container status \"bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf\": rpc error: code = NotFound desc = could not find container \"bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf\": container with ID starting with bfb2e11e33dcd6233c81bc72296c5b58dc7936ca264bb8a74565761e3bb26ecf not found: ID does not exist"
Sep 07 22:47:20.470950 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.470919    2582 scope.go:117] "RemoveContainer" containerID="d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7"
Sep 07 22:47:20.471172 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:47:20.471154    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7\": container with ID starting with d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7 not found: ID does not exist" containerID="d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7"
Sep 07 22:47:20.471228 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.471182    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7"} err="failed to get container status \"d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7\": rpc error: code = NotFound desc = could not find container \"d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7\": container with ID starting with d27dba41e24233dfd40996d85fbb77c9c72cb98a7e3e3ebad28914c3888dd3b7 not found: ID does not exist"
Sep 07 22:47:20.478931 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.478907    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"]
Sep 07 22:47:20.481920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:20.481894    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-66774a4a-openshift-default-6dff47b544-grmpd"]
Sep 07 22:47:21.459958 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:21.459914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2" event={"ID":"57df0612-bde2-428a-95bb-5d92e92213a0","Type":"ContainerStarted","Data":"2fb722ea4e4f4d956e3eb6617d65314a82957701ff39c91c3de421f31a3181c9"}
Sep 07 22:47:21.460427 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:21.460098    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:21.478197 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:21.478136    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2" podStartSLOduration=3.118373585 podStartE2EDuration="3.478113806s" totalImagesPullingTime="359.740221ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:47:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:47:21.477498903 +0000 UTC m=+907.943655926" watchObservedRunningTime="2026-09-07 22:47:21.478113806 +0000 UTC m=+907.944270803"
Sep 07 22:47:22.094694 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:22.094657    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" path="/var/lib/kubelet/pods/7cfd15bc-8865-4659-bbcc-f95beac7c810/volumes"
Sep 07 22:47:26.198923 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:26.198885    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"]
Sep 07 22:47:26.200999 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:26.200975    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-7c3369-75pjv"]
Sep 07 22:47:27.472370 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:27.472333    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:28.089978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:28.089939    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bee87d33-75cb-4a3c-b800-1e9f86e9fc4d" path="/var/lib/kubelet/pods/bee87d33-75cb-4a3c-b800-1e9f86e9fc4d/volumes"
Sep 07 22:47:31.464079 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:31.464047    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:49.396953 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.396914    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"]
Sep 07 22:47:49.397645 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.397146    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2" podUID="57df0612-bde2-428a-95bb-5d92e92213a0" containerName="maas-api" containerID="cri-o://2fb722ea4e4f4d956e3eb6617d65314a82957701ff39c91c3de421f31a3181c9" gracePeriod=30
Sep 07 22:47:49.454299 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.454244    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"]
Sep 07 22:47:49.454564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.454530    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw" podUID="e1b21894-98ab-4c9a-82dd-1916d498c834" containerName="payload-pre-processing" containerID="cri-o://d6cef2a97ea97ab47d9e22c6dd54354a7d0ff5a4d9a374b9cb4dad84501d8a4a" gracePeriod=30
Sep 07 22:47:49.624692 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.623688    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1b21894-98ab-4c9a-82dd-1916d498c834" containerID="d6cef2a97ea97ab47d9e22c6dd54354a7d0ff5a4d9a374b9cb4dad84501d8a4a" exitCode=0
Sep 07 22:47:49.624692 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.623774    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw" event={"ID":"e1b21894-98ab-4c9a-82dd-1916d498c834","Type":"ContainerDied","Data":"d6cef2a97ea97ab47d9e22c6dd54354a7d0ff5a4d9a374b9cb4dad84501d8a4a"}
Sep 07 22:47:49.626031 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.625905    2582 generic.go:356] "Generic (PLEG): container finished" podID="57df0612-bde2-428a-95bb-5d92e92213a0" containerID="2fb722ea4e4f4d956e3eb6617d65314a82957701ff39c91c3de421f31a3181c9" exitCode=0
Sep 07 22:47:49.626031 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.625942    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2" event={"ID":"57df0612-bde2-428a-95bb-5d92e92213a0","Type":"ContainerDied","Data":"2fb722ea4e4f4d956e3eb6617d65314a82957701ff39c91c3de421f31a3181c9"}
Sep 07 22:47:49.727694 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.727651    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:49.757719 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.757684    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:49.826116 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.826084    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls\") pod \"57df0612-bde2-428a-95bb-5d92e92213a0\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") "
Sep 07 22:47:49.826324 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.826145    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2bnv9\" (UniqueName: \"kubernetes.io/projected/57df0612-bde2-428a-95bb-5d92e92213a0-kube-api-access-2bnv9\") pod \"57df0612-bde2-428a-95bb-5d92e92213a0\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") "
Sep 07 22:47:49.826324 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.826221    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-metrics-certs\") pod \"57df0612-bde2-428a-95bb-5d92e92213a0\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") "
Sep 07 22:47:49.826324 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.826270    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57df0612-bde2-428a-95bb-5d92e92213a0-openshift-service-ca\") pod \"57df0612-bde2-428a-95bb-5d92e92213a0\" (UID: \"57df0612-bde2-428a-95bb-5d92e92213a0\") "
Sep 07 22:47:49.827074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.827042    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57df0612-bde2-428a-95bb-5d92e92213a0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "57df0612-bde2-428a-95bb-5d92e92213a0" (UID: "57df0612-bde2-428a-95bb-5d92e92213a0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:49.828584 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.828557    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "57df0612-bde2-428a-95bb-5d92e92213a0" (UID: "57df0612-bde2-428a-95bb-5d92e92213a0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:47:49.828687 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.828628    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "57df0612-bde2-428a-95bb-5d92e92213a0" (UID: "57df0612-bde2-428a-95bb-5d92e92213a0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:47:49.828687 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.828674    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57df0612-bde2-428a-95bb-5d92e92213a0-kube-api-access-2bnv9" (OuterVolumeSpecName: "kube-api-access-2bnv9") pod "57df0612-bde2-428a-95bb-5d92e92213a0" (UID: "57df0612-bde2-428a-95bb-5d92e92213a0"). InnerVolumeSpecName "kube-api-access-2bnv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:49.927567 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.927462    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bgkcb\" (UniqueName: \"kubernetes.io/projected/e1b21894-98ab-4c9a-82dd-1916d498c834-kube-api-access-bgkcb\") pod \"e1b21894-98ab-4c9a-82dd-1916d498c834\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") "
Sep 07 22:47:49.927749 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.927576    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b21894-98ab-4c9a-82dd-1916d498c834-plugins-config-volume\") pod \"e1b21894-98ab-4c9a-82dd-1916d498c834\" (UID: \"e1b21894-98ab-4c9a-82dd-1916d498c834\") "
Sep 07 22:47:49.927818 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.927778    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bnv9\" (UniqueName: \"kubernetes.io/projected/57df0612-bde2-428a-95bb-5d92e92213a0-kube-api-access-2bnv9\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:49.927818 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.927790    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:49.927818 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.927799    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/57df0612-bde2-428a-95bb-5d92e92213a0-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:49.927818 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.927809    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/57df0612-bde2-428a-95bb-5d92e92213a0-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:49.928021 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.927932    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1b21894-98ab-4c9a-82dd-1916d498c834-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1b21894-98ab-4c9a-82dd-1916d498c834" (UID: "e1b21894-98ab-4c9a-82dd-1916d498c834"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:47:49.929613 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:49.929591    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1b21894-98ab-4c9a-82dd-1916d498c834-kube-api-access-bgkcb" (OuterVolumeSpecName: "kube-api-access-bgkcb") pod "e1b21894-98ab-4c9a-82dd-1916d498c834" (UID: "e1b21894-98ab-4c9a-82dd-1916d498c834"). InnerVolumeSpecName "kube-api-access-bgkcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:47:50.029080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.029039    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bgkcb\" (UniqueName: \"kubernetes.io/projected/e1b21894-98ab-4c9a-82dd-1916d498c834-kube-api-access-bgkcb\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:50.029080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.029073    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1b21894-98ab-4c9a-82dd-1916d498c834-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:47:50.633195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.633147    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2" event={"ID":"57df0612-bde2-428a-95bb-5d92e92213a0","Type":"ContainerDied","Data":"fe1c3b122672a6b565e44e07e0d2f3766c3c9dfd30fb6493483e2c646d34d78b"}
Sep 07 22:47:50.633661 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.633242    2582 scope.go:117] "RemoveContainer" containerID="2fb722ea4e4f4d956e3eb6617d65314a82957701ff39c91c3de421f31a3181c9"
Sep 07 22:47:50.633661 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.633159    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"
Sep 07 22:47:50.634974 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.634948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw" event={"ID":"e1b21894-98ab-4c9a-82dd-1916d498c834","Type":"ContainerDied","Data":"5187f1ffe2b209752dfda1a7eab8112bd6f1e5c680fc8f369daea5a42ed106f8"}
Sep 07 22:47:50.635103 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.634962    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"
Sep 07 22:47:50.643279 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.643257    2582 scope.go:117] "RemoveContainer" containerID="d6cef2a97ea97ab47d9e22c6dd54354a7d0ff5a4d9a374b9cb4dad84501d8a4a"
Sep 07 22:47:50.652335 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.652310    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"]
Sep 07 22:47:50.657690 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.657664    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-01ae3268-d8d4bcf47-8pxn2"]
Sep 07 22:47:50.667690 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.667619    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"]
Sep 07 22:47:50.674015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:50.673993    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-01ae3268-659d988774-x22nw"]
Sep 07 22:47:52.093956 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:52.093915    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57df0612-bde2-428a-95bb-5d92e92213a0" path="/var/lib/kubelet/pods/57df0612-bde2-428a-95bb-5d92e92213a0/volumes"
Sep 07 22:47:52.094364 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:47:52.094335    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1b21894-98ab-4c9a-82dd-1916d498c834" path="/var/lib/kubelet/pods/e1b21894-98ab-4c9a-82dd-1916d498c834/volumes"
Sep 07 22:48:08.710862 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:08.710826    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"]
Sep 07 22:48:08.711310 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:08.711054    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy" containerID="cri-o://0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969" gracePeriod=30
Sep 07 22:48:13.772174 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772137    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"]
Sep 07 22:48:13.772707 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772686    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy"
Sep 07 22:48:13.772799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772709    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy"
Sep 07 22:48:13.772799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772727    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy"
Sep 07 22:48:13.772799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772736    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy"
Sep 07 22:48:13.772799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772761    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57df0612-bde2-428a-95bb-5d92e92213a0" containerName="maas-api"
Sep 07 22:48:13.772799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772769    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="57df0612-bde2-428a-95bb-5d92e92213a0" containerName="maas-api"
Sep 07 22:48:13.772799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772789    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1b21894-98ab-4c9a-82dd-1916d498c834" containerName="payload-pre-processing"
Sep 07 22:48:13.772799 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772797    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1b21894-98ab-4c9a-82dd-1916d498c834" containerName="payload-pre-processing"
Sep 07 22:48:13.773131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772896    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy"
Sep 07 22:48:13.773131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772908    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cfd15bc-8865-4659-bbcc-f95beac7c810" containerName="istio-proxy"
Sep 07 22:48:13.773131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772920    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="57df0612-bde2-428a-95bb-5d92e92213a0" containerName="maas-api"
Sep 07 22:48:13.773131 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.772933    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1b21894-98ab-4c9a-82dd-1916d498c834" containerName="payload-pre-processing"
Sep 07 22:48:13.776137 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.776114    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.778498 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.778471    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:48:13.778625 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.778515    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-52b103-serving-cert\""
Sep 07 22:48:13.779517 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.779494    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 22:48:13.779707 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.779649    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 22:48:13.779844 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.779781    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 22:48:13.788792 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.788725    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"]
Sep 07 22:48:13.847051 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.847018    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6494z\" (UniqueName: \"kubernetes.io/projected/a80db81d-425f-4884-bffc-a517f8d9f795-kube-api-access-6494z\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.847195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.847061    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.847195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.847128    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80db81d-425f-4884-bffc-a517f8d9f795-openshift-service-ca\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.847322 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.847217    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-metrics-certs\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.948272 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.948239    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-metrics-certs\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.948410 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.948309    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6494z\" (UniqueName: \"kubernetes.io/projected/a80db81d-425f-4884-bffc-a517f8d9f795-kube-api-access-6494z\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.948410 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.948338    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.948410 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.948378    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80db81d-425f-4884-bffc-a517f8d9f795-openshift-service-ca\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.950781 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.950754    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-explicit-52b103-serving-cert\""
Sep 07 22:48:13.950889 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.950756    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:48:13.950889 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.950769    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 22:48:13.956385 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.956365    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 22:48:13.958558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.958540    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:48:13.958663 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:48:13.958645    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-serving-cert: secret "maas-api-e2e-explicit-52b103-serving-cert" not found
Sep 07 22:48:13.958743 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:48:13.958731    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls podName:a80db81d-425f-4884-bffc-a517f8d9f795 nodeName:}" failed. No retries permitted until 2026-09-07 22:48:14.458708002 +0000 UTC m=+960.924864993 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls") pod "maas-api-e2e-explicit-52b103-7595786d87-w8p4c" (UID: "a80db81d-425f-4884-bffc-a517f8d9f795") : secret "maas-api-e2e-explicit-52b103-serving-cert" not found
Sep 07 22:48:13.959106 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.959090    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80db81d-425f-4884-bffc-a517f8d9f795-openshift-service-ca\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.960970 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.960953    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-metrics-certs\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:13.968716 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:13.968684    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6494z\" (UniqueName: \"kubernetes.io/projected/a80db81d-425f-4884-bffc-a517f8d9f795-kube-api-access-6494z\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:14.049190 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049163    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-envoy\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049278 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049206    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-socket\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049278 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049236    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-credential-socket\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049278 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049267    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-data\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049465 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049329    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tc9ch\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-kube-api-access-tc9ch\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049465 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049349    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f54d5-c723-4dc7-b565-54db8e10122c-istio-podinfo\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049465 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049373    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-istio-token\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049465 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049420    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f54d5-c723-4dc7-b565-54db8e10122c-istiod-ca-cert\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049524    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-certs\") pod \"569f54d5-c723-4dc7-b565-54db8e10122c\" (UID: \"569f54d5-c723-4dc7-b565-54db8e10122c\") "
Sep 07 22:48:14.049801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049611    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:14.049801 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049684    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:14.050028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049830    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-data" (OuterVolumeSpecName: "istio-data") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:14.050028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049930    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.050028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049951    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-credential-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.050028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.049980    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-data\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.050249 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.050119    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:14.050341 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.050312    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/569f54d5-c723-4dc7-b565-54db8e10122c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:48:14.051896 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.051864    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-istio-token" (OuterVolumeSpecName: "istio-token") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:48:14.052019 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.051964    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/569f54d5-c723-4dc7-b565-54db8e10122c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:48:14.052094 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.052029    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:48:14.052207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.052180    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-kube-api-access-tc9ch" (OuterVolumeSpecName: "kube-api-access-tc9ch") pod "569f54d5-c723-4dc7-b565-54db8e10122c" (UID: "569f54d5-c723-4dc7-b565-54db8e10122c"). InnerVolumeSpecName "kube-api-access-tc9ch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:48:14.151069 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.151029    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc9ch\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-kube-api-access-tc9ch\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.151069 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.151064    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/569f54d5-c723-4dc7-b565-54db8e10122c-istio-podinfo\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.151069 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.151074    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/569f54d5-c723-4dc7-b565-54db8e10122c-istio-token\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.151335 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.151082    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/569f54d5-c723-4dc7-b565-54db8e10122c-istiod-ca-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.151335 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.151090    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-workload-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.151335 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.151098    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/569f54d5-c723-4dc7-b565-54db8e10122c-istio-envoy\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:48:14.410006 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.409970    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"]
Sep 07 22:48:14.410410 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.410396    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy"
Sep 07 22:48:14.410478 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.410411    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy"
Sep 07 22:48:14.410478 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.410421    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy"
Sep 07 22:48:14.410478 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.410427    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy"
Sep 07 22:48:14.410590 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.410498    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy"
Sep 07 22:48:14.410590 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.410504    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" containerName="istio-proxy"
Sep 07 22:48:14.414787 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.414763    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.417249 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.417223    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-52b103-dockercfg-bnfqz\""
Sep 07 22:48:14.417505 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.417486    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-52b103\""
Sep 07 22:48:14.425822 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.425792    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"]
Sep 07 22:48:14.454342 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.454272    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjl7x\" (UniqueName: \"kubernetes.io/projected/20c7ec13-39d6-4971-8248-fbf77383762a-kube-api-access-hjl7x\") pod \"payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") " pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.454534 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.454418    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c7ec13-39d6-4971-8248-fbf77383762a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") " pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.556014 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.555966    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjl7x\" (UniqueName: \"kubernetes.io/projected/20c7ec13-39d6-4971-8248-fbf77383762a-kube-api-access-hjl7x\") pod \"payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") " pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.556197 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.556111    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c7ec13-39d6-4971-8248-fbf77383762a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") " pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.556197 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.556148    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:14.556871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.556842    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c7ec13-39d6-4971-8248-fbf77383762a-plugins-config-volume\") pod \"payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") " pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.558545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.558523    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls\") pod \"maas-api-e2e-explicit-52b103-7595786d87-w8p4c\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") " pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:14.572510 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.572479    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjl7x\" (UniqueName: \"kubernetes.io/projected/20c7ec13-39d6-4971-8248-fbf77383762a-kube-api-access-hjl7x\") pod \"payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") " pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.699162 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.699079    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 22:48:14.707636 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.707602    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:14.726989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.726941    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:14.777510 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.777469    2582 generic.go:356] "Generic (PLEG): container finished" podID="569f54d5-c723-4dc7-b565-54db8e10122c" containerID="0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969" exitCode=0
Sep 07 22:48:14.777907 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.777543    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" event={"ID":"569f54d5-c723-4dc7-b565-54db8e10122c","Type":"ContainerDied","Data":"0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969"}
Sep 07 22:48:14.777907 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.777597    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j" event={"ID":"569f54d5-c723-4dc7-b565-54db8e10122c","Type":"ContainerDied","Data":"2653ea6c90910f9aee738e28bab9c2c28640c50a0616ae9f57efc943c5db48f9"}
Sep 07 22:48:14.777907 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.777634    2582 scope.go:117] "RemoveContainer" containerID="0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969"
Sep 07 22:48:14.777907 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.777839    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"
Sep 07 22:48:14.791233 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.791210    2582 scope.go:117] "RemoveContainer" containerID="7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b"
Sep 07 22:48:14.798452 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.798275    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"]
Sep 07 22:48:14.801494 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.801466    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-autores-7c3369-openshift-default-674b75f5b-ws78j"]
Sep 07 22:48:14.806107 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.806085    2582 scope.go:117] "RemoveContainer" containerID="0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969"
Sep 07 22:48:14.806557 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:48:14.806526    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969\": container with ID starting with 0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969 not found: ID does not exist" containerID="0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969"
Sep 07 22:48:14.806656 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.806570    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969"} err="failed to get container status \"0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969\": rpc error: code = NotFound desc = could not find container \"0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969\": container with ID starting with 0f782d60c02afe7be89f09ae1f3bc9d81173905b67bcb24a0a931b315ad27969 not found: ID does not exist"
Sep 07 22:48:14.806656 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.806598    2582 scope.go:117] "RemoveContainer" containerID="7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b"
Sep 07 22:48:14.806975 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:48:14.806930    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b\": container with ID starting with 7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b not found: ID does not exist" containerID="7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b"
Sep 07 22:48:14.807036 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.806985    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b"} err="failed to get container status \"7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b\": rpc error: code = NotFound desc = could not find container \"7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b\": container with ID starting with 7f9dae4cf42d2e38bea1495885ba1811151a08b41868fa7b9eea85674124535b not found: ID does not exist"
Sep 07 22:48:14.981582 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:14.981546    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"]
Sep 07 22:48:14.982407 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:48:14.982379    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda80db81d_425f_4884_bffc_a517f8d9f795.slice/crio-3b27902716c30c45e80f6e1fb0197cf5673535634c0fb58d1575c952e437b938 WatchSource:0}: Error finding container 3b27902716c30c45e80f6e1fb0197cf5673535634c0fb58d1575c952e437b938: Status 404 returned error can't find the container with id 3b27902716c30c45e80f6e1fb0197cf5673535634c0fb58d1575c952e437b938
Sep 07 22:48:15.021203 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.021177    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"]
Sep 07 22:48:15.023788 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:48:15.023757    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c7ec13_39d6_4971_8248_fbf77383762a.slice/crio-40f1d1e1b6e9373c8e614412b4b02fa88cb6302c75afce0b2d8778c24e1975a3 WatchSource:0}: Error finding container 40f1d1e1b6e9373c8e614412b4b02fa88cb6302c75afce0b2d8778c24e1975a3: Status 404 returned error can't find the container with id 40f1d1e1b6e9373c8e614412b4b02fa88cb6302c75afce0b2d8778c24e1975a3
Sep 07 22:48:15.786239 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.786139    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c" event={"ID":"a80db81d-425f-4884-bffc-a517f8d9f795","Type":"ContainerStarted","Data":"81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84"}
Sep 07 22:48:15.786239 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.786199    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c" event={"ID":"a80db81d-425f-4884-bffc-a517f8d9f795","Type":"ContainerStarted","Data":"3b27902716c30c45e80f6e1fb0197cf5673535634c0fb58d1575c952e437b938"}
Sep 07 22:48:15.786789 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.786319    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:15.787949 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.787923    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv" event={"ID":"20c7ec13-39d6-4971-8248-fbf77383762a","Type":"ContainerStarted","Data":"30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056"}
Sep 07 22:48:15.788080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.787968    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv" event={"ID":"20c7ec13-39d6-4971-8248-fbf77383762a","Type":"ContainerStarted","Data":"40f1d1e1b6e9373c8e614412b4b02fa88cb6302c75afce0b2d8778c24e1975a3"}
Sep 07 22:48:15.788080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.787998    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:48:15.802986 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:15.802937    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c" podStartSLOduration=2.403199231 podStartE2EDuration="2.802922573s" totalImagesPullingTime="399.723342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:48:15.801123402 +0000 UTC m=+962.267280410" watchObservedRunningTime="2026-09-07 22:48:15.802922573 +0000 UTC m=+962.269079568"
Sep 07 22:48:16.090046 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:16.090009    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="569f54d5-c723-4dc7-b565-54db8e10122c" path="/var/lib/kubelet/pods/569f54d5-c723-4dc7-b565-54db8e10122c/volumes"
Sep 07 22:48:21.800335 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:21.800303    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:48:21.817457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:21.817389    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv" podStartSLOduration=7.817367396 podStartE2EDuration="7.817367396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:48:15.818563525 +0000 UTC m=+962.284720521" watchObservedRunningTime="2026-09-07 22:48:21.817367396 +0000 UTC m=+968.283524391"
Sep 07 22:48:26.797149 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:48:26.797116    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:49:05.871057 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.871014    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"]
Sep 07 22:49:05.875271 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.875238    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:05.877665 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.877634    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-52b103\"/\"default-dockercfg-fknf4\""
Sep 07 22:49:05.878696 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.878671    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-52b103\"/\"openshift-service-ca.crt\""
Sep 07 22:49:05.878696 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.878686    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-52b103\"/\"explicit-model-52b103-kserve-self-signed-certs\""
Sep 07 22:49:05.878858 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.878743    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-52b103\"/\"kube-root-ca.crt\""
Sep 07 22:49:05.885022 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.884571    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"]
Sep 07 22:49:05.954687 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.954637    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tls-certs\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:05.954876 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.954722    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tmp-dir\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:05.954876 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.954747    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-model-cache\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:05.954876 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.954772    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-dshm\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:05.954876 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.954792    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bvb7\" (UniqueName: \"kubernetes.io/projected/2a4abc76-8b39-47f7-b6a2-50b701cc1924-kube-api-access-6bvb7\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:05.955041 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:05.954865    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-home\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.055484 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.055448    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tmp-dir\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.055484 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.055490    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-model-cache\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.055757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.055518    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-dshm\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.055757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.055543    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bvb7\" (UniqueName: \"kubernetes.io/projected/2a4abc76-8b39-47f7-b6a2-50b701cc1924-kube-api-access-6bvb7\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.055757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.055588    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-home\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.055757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.055735    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tls-certs\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.055971 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.055923    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tmp-dir\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.056028 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.056008    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-home\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.056233 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.056214    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-model-cache\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.058067 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.058044    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-dshm\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.058209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.058188    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tls-certs\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.062698 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.062675    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bvb7\" (UniqueName: \"kubernetes.io/projected/2a4abc76-8b39-47f7-b6a2-50b701cc1924-kube-api-access-6bvb7\") pod \"explicit-model-52b103-kserve-6745b45859-84nq7\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") " pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.187677 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.187581    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:06.472944 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:06.472910    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"]
Sep 07 22:49:06.474205 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:49:06.474178    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a4abc76_8b39_47f7_b6a2_50b701cc1924.slice/crio-a424ac284b8dc68a1a9fb15b1b27ac484fd54cf824ae98e156c90cceabfb98cd WatchSource:0}: Error finding container a424ac284b8dc68a1a9fb15b1b27ac484fd54cf824ae98e156c90cceabfb98cd: Status 404 returned error can't find the container with id a424ac284b8dc68a1a9fb15b1b27ac484fd54cf824ae98e156c90cceabfb98cd
Sep 07 22:49:07.082504 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:07.082467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7" event={"ID":"2a4abc76-8b39-47f7-b6a2-50b701cc1924","Type":"ContainerStarted","Data":"7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676"}
Sep 07 22:49:07.082908 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:07.082518    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7" event={"ID":"2a4abc76-8b39-47f7-b6a2-50b701cc1924","Type":"ContainerStarted","Data":"a424ac284b8dc68a1a9fb15b1b27ac484fd54cf824ae98e156c90cceabfb98cd"}
Sep 07 22:49:07.096797 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:07.096729    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7" podStartSLOduration=2.096707274 podStartE2EDuration="2.096707274s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:07.096338606 +0000 UTC m=+1013.562495599" watchObservedRunningTime="2026-09-07 22:49:07.096707274 +0000 UTC m=+1013.562864270"
Sep 07 22:49:16.187921 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:16.187854    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:16.188371 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:16.187942    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:16.200978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:16.200946    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:17.148239 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:17.148209    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:19.048075 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:19.048040    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"]
Sep 07 22:49:28.916273 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:28.916232    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"]
Sep 07 22:49:28.921155 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:28.921133    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:28.923995 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:28.923974    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-2872fd39\""
Sep 07 22:49:28.924119 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:28.923980    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-2872fd39-dockercfg-85649\""
Sep 07 22:49:28.930973 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:28.930942    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"]
Sep 07 22:49:28.978930 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:28.978894    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:28.979104 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:28.978972    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrjc2\" (UniqueName: \"kubernetes.io/projected/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-kube-api-access-jrjc2\") pod \"payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:29.080136 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.080075    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:29.080386 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.080157    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrjc2\" (UniqueName: \"kubernetes.io/projected/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-kube-api-access-jrjc2\") pod \"payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:29.080830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.080807    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:29.088014 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.087979    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrjc2\" (UniqueName: \"kubernetes.io/projected/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-kube-api-access-jrjc2\") pod \"payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:29.232221 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.232127    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:29.504080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.504044    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"]
Sep 07 22:49:29.508209 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:49:29.508183    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda63bdbe3_0058_47a5_879a_1a15e5fd1fd8.slice/crio-c1cfb250c17b65feb62c8ee306ab5fbdd34a567fe5508ea94928e6a885a0f897 WatchSource:0}: Error finding container c1cfb250c17b65feb62c8ee306ab5fbdd34a567fe5508ea94928e6a885a0f897: Status 404 returned error can't find the container with id c1cfb250c17b65feb62c8ee306ab5fbdd34a567fe5508ea94928e6a885a0f897
Sep 07 22:49:29.660881 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.660843    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"]
Sep 07 22:49:29.661470 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.661098    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c" podUID="a80db81d-425f-4884-bffc-a517f8d9f795" containerName="maas-api" containerID="cri-o://81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84" gracePeriod=30
Sep 07 22:49:29.710195 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.710162    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"]
Sep 07 22:49:29.710477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.710450    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv" podUID="20c7ec13-39d6-4971-8248-fbf77383762a" containerName="payload-processing" containerID="cri-o://30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056" gracePeriod=30
Sep 07 22:49:29.963975 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.963887    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:49:29.995850 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:29.995824    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:49:30.091712 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.091630    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-metrics-certs\") pod \"a80db81d-425f-4884-bffc-a517f8d9f795\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") "
Sep 07 22:49:30.091712 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.091676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c7ec13-39d6-4971-8248-fbf77383762a-plugins-config-volume\") pod \"20c7ec13-39d6-4971-8248-fbf77383762a\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") "
Sep 07 22:49:30.091949 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.091737    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls\") pod \"a80db81d-425f-4884-bffc-a517f8d9f795\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") "
Sep 07 22:49:30.091949 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.091797    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80db81d-425f-4884-bffc-a517f8d9f795-openshift-service-ca\") pod \"a80db81d-425f-4884-bffc-a517f8d9f795\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") "
Sep 07 22:49:30.092151 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.092131    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6494z\" (UniqueName: \"kubernetes.io/projected/a80db81d-425f-4884-bffc-a517f8d9f795-kube-api-access-6494z\") pod \"a80db81d-425f-4884-bffc-a517f8d9f795\" (UID: \"a80db81d-425f-4884-bffc-a517f8d9f795\") "
Sep 07 22:49:30.092321 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.092304    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjl7x\" (UniqueName: \"kubernetes.io/projected/20c7ec13-39d6-4971-8248-fbf77383762a-kube-api-access-hjl7x\") pod \"20c7ec13-39d6-4971-8248-fbf77383762a\" (UID: \"20c7ec13-39d6-4971-8248-fbf77383762a\") "
Sep 07 22:49:30.092531 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.092356    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a80db81d-425f-4884-bffc-a517f8d9f795-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a80db81d-425f-4884-bffc-a517f8d9f795" (UID: "a80db81d-425f-4884-bffc-a517f8d9f795"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:49:30.092649 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.092623    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c7ec13-39d6-4971-8248-fbf77383762a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "20c7ec13-39d6-4971-8248-fbf77383762a" (UID: "20c7ec13-39d6-4971-8248-fbf77383762a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:49:30.092957 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.092935    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c7ec13-39d6-4971-8248-fbf77383762a-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.093059 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.092975    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a80db81d-425f-4884-bffc-a517f8d9f795-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.094779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.094743    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a80db81d-425f-4884-bffc-a517f8d9f795" (UID: "a80db81d-425f-4884-bffc-a517f8d9f795"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:49:30.095147 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.094985    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c7ec13-39d6-4971-8248-fbf77383762a-kube-api-access-hjl7x" (OuterVolumeSpecName: "kube-api-access-hjl7x") pod "20c7ec13-39d6-4971-8248-fbf77383762a" (UID: "20c7ec13-39d6-4971-8248-fbf77383762a"). InnerVolumeSpecName "kube-api-access-hjl7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:49:30.095795 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.095780    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a80db81d-425f-4884-bffc-a517f8d9f795" (UID: "a80db81d-425f-4884-bffc-a517f8d9f795"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:49:30.096366 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.096340    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a80db81d-425f-4884-bffc-a517f8d9f795-kube-api-access-6494z" (OuterVolumeSpecName: "kube-api-access-6494z") pod "a80db81d-425f-4884-bffc-a517f8d9f795" (UID: "a80db81d-425f-4884-bffc-a517f8d9f795"). InnerVolumeSpecName "kube-api-access-6494z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:49:30.194103 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.194067    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6494z\" (UniqueName: \"kubernetes.io/projected/a80db81d-425f-4884-bffc-a517f8d9f795-kube-api-access-6494z\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.194103 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.194097    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjl7x\" (UniqueName: \"kubernetes.io/projected/20c7ec13-39d6-4971-8248-fbf77383762a-kube-api-access-hjl7x\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.194103 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.194107    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.194348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.194116    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a80db81d-425f-4884-bffc-a517f8d9f795-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:30.210687 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.210652    2582 generic.go:356] "Generic (PLEG): container finished" podID="20c7ec13-39d6-4971-8248-fbf77383762a" containerID="30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056" exitCode=0
Sep 07 22:49:30.210837 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.210691    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"
Sep 07 22:49:30.210837 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.210736    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv" event={"ID":"20c7ec13-39d6-4971-8248-fbf77383762a","Type":"ContainerDied","Data":"30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056"}
Sep 07 22:49:30.210837 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.210787    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv" event={"ID":"20c7ec13-39d6-4971-8248-fbf77383762a","Type":"ContainerDied","Data":"40f1d1e1b6e9373c8e614412b4b02fa88cb6302c75afce0b2d8778c24e1975a3"}
Sep 07 22:49:30.210837 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.210817    2582 scope.go:117] "RemoveContainer" containerID="30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056"
Sep 07 22:49:30.212635 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.212609    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955" event={"ID":"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8","Type":"ContainerStarted","Data":"d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97"}
Sep 07 22:49:30.212770 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.212655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955" event={"ID":"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8","Type":"ContainerStarted","Data":"c1cfb250c17b65feb62c8ee306ab5fbdd34a567fe5508ea94928e6a885a0f897"}
Sep 07 22:49:30.212999 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.212964    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:30.214335 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.214278    2582 generic.go:356] "Generic (PLEG): container finished" podID="a80db81d-425f-4884-bffc-a517f8d9f795" containerID="81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84" exitCode=0
Sep 07 22:49:30.214443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.214400    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c" event={"ID":"a80db81d-425f-4884-bffc-a517f8d9f795","Type":"ContainerDied","Data":"81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84"}
Sep 07 22:49:30.214443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.214421    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"
Sep 07 22:49:30.214443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.214440    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c" event={"ID":"a80db81d-425f-4884-bffc-a517f8d9f795","Type":"ContainerDied","Data":"3b27902716c30c45e80f6e1fb0197cf5673535634c0fb58d1575c952e437b938"}
Sep 07 22:49:30.222479 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.222463    2582 scope.go:117] "RemoveContainer" containerID="30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056"
Sep 07 22:49:30.222741 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:49:30.222723    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056\": container with ID starting with 30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056 not found: ID does not exist" containerID="30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056"
Sep 07 22:49:30.222810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.222749    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056"} err="failed to get container status \"30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056\": rpc error: code = NotFound desc = could not find container \"30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056\": container with ID starting with 30f38545f27c1bb843a448b1d80e8014ad3093ff33109dd3e0f0c2b2b928c056 not found: ID does not exist"
Sep 07 22:49:30.222810 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.222764    2582 scope.go:117] "RemoveContainer" containerID="81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84"
Sep 07 22:49:30.231073 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.231031    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955" podStartSLOduration=2.231017421 podStartE2EDuration="2.231017421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:30.228068821 +0000 UTC m=+1036.694225817" watchObservedRunningTime="2026-09-07 22:49:30.231017421 +0000 UTC m=+1036.697174416"
Sep 07 22:49:30.231871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.231848    2582 scope.go:117] "RemoveContainer" containerID="81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84"
Sep 07 22:49:30.232125 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:49:30.232106    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84\": container with ID starting with 81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84 not found: ID does not exist" containerID="81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84"
Sep 07 22:49:30.232193 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.232137    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84"} err="failed to get container status \"81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84\": rpc error: code = NotFound desc = could not find container \"81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84\": container with ID starting with 81d9fe4d365622a9160c06a562e0d9a9f087e846915e267322fb14c73d7e0a84 not found: ID does not exist"
Sep 07 22:49:30.242192 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.242170    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"]
Sep 07 22:49:30.245851 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.245831    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-explicit-52b103-7595786d87-w8p4c"]
Sep 07 22:49:30.258154 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.258129    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"]
Sep 07 22:49:30.261423 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:30.261400    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-52b103-655f5d4985-xq6qv"]
Sep 07 22:49:32.095875 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:32.095836    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="20c7ec13-39d6-4971-8248-fbf77383762a" path="/var/lib/kubelet/pods/20c7ec13-39d6-4971-8248-fbf77383762a/volumes"
Sep 07 22:49:32.096253 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:32.096220    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a80db81d-425f-4884-bffc-a517f8d9f795" path="/var/lib/kubelet/pods/a80db81d-425f-4884-bffc-a517f8d9f795/volumes"
Sep 07 22:49:34.170245 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.170203    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7" podUID="2a4abc76-8b39-47f7-b6a2-50b701cc1924" containerName="main" containerID="cri-o://7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676" gracePeriod=45
Sep 07 22:49:34.437182 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.437152    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:34.534624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.534586    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-model-cache\") pod \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") "
Sep 07 22:49:34.534773 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.534637    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-dshm\") pod \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") "
Sep 07 22:49:34.534773 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.534666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bvb7\" (UniqueName: \"kubernetes.io/projected/2a4abc76-8b39-47f7-b6a2-50b701cc1924-kube-api-access-6bvb7\") pod \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") "
Sep 07 22:49:34.534773 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.534695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tls-certs\") pod \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") "
Sep 07 22:49:34.534773 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.534758    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tmp-dir\") pod \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") "
Sep 07 22:49:34.535009 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.534786    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-home\") pod \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\" (UID: \"2a4abc76-8b39-47f7-b6a2-50b701cc1924\") "
Sep 07 22:49:34.535009 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.534913    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a4abc76-8b39-47f7-b6a2-50b701cc1924" (UID: "2a4abc76-8b39-47f7-b6a2-50b701cc1924"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:34.535121 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.535036    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a4abc76-8b39-47f7-b6a2-50b701cc1924" (UID: "2a4abc76-8b39-47f7-b6a2-50b701cc1924"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:34.535121 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.535094    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-home" (OuterVolumeSpecName: "home") pod "2a4abc76-8b39-47f7-b6a2-50b701cc1924" (UID: "2a4abc76-8b39-47f7-b6a2-50b701cc1924"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:34.535206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.535183    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tmp-dir\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:34.535206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.535201    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-home\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:34.535306 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.535215    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-model-cache\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:34.536953 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.536925    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-dshm" (OuterVolumeSpecName: "dshm") pod "2a4abc76-8b39-47f7-b6a2-50b701cc1924" (UID: "2a4abc76-8b39-47f7-b6a2-50b701cc1924"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:49:34.537069 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.536977    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a4abc76-8b39-47f7-b6a2-50b701cc1924" (UID: "2a4abc76-8b39-47f7-b6a2-50b701cc1924"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:49:34.537069 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.537003    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a4abc76-8b39-47f7-b6a2-50b701cc1924-kube-api-access-6bvb7" (OuterVolumeSpecName: "kube-api-access-6bvb7") pod "2a4abc76-8b39-47f7-b6a2-50b701cc1924" (UID: "2a4abc76-8b39-47f7-b6a2-50b701cc1924"). InnerVolumeSpecName "kube-api-access-6bvb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:49:34.636457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.636413    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a4abc76-8b39-47f7-b6a2-50b701cc1924-dshm\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:34.636457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.636449    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bvb7\" (UniqueName: \"kubernetes.io/projected/2a4abc76-8b39-47f7-b6a2-50b701cc1924-kube-api-access-6bvb7\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:34.636457 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:34.636465    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a4abc76-8b39-47f7-b6a2-50b701cc1924-tls-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:49:35.245035 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.244997    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a4abc76-8b39-47f7-b6a2-50b701cc1924" containerID="7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676" exitCode=0
Sep 07 22:49:35.245507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.245042    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"
Sep 07 22:49:35.245507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.245084    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7" event={"ID":"2a4abc76-8b39-47f7-b6a2-50b701cc1924","Type":"ContainerDied","Data":"7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676"}
Sep 07 22:49:35.245507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.245132    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7" event={"ID":"2a4abc76-8b39-47f7-b6a2-50b701cc1924","Type":"ContainerDied","Data":"a424ac284b8dc68a1a9fb15b1b27ac484fd54cf824ae98e156c90cceabfb98cd"}
Sep 07 22:49:35.245507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.245155    2582 scope.go:117] "RemoveContainer" containerID="7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676"
Sep 07 22:49:35.254190 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.254170    2582 scope.go:117] "RemoveContainer" containerID="7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676"
Sep 07 22:49:35.254512 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:49:35.254494    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676\": container with ID starting with 7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676 not found: ID does not exist" containerID="7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676"
Sep 07 22:49:35.254568 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.254521    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676"} err="failed to get container status \"7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676\": rpc error: code = NotFound desc = could not find container \"7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676\": container with ID starting with 7676f788ad68d8434fa6f26de8e3a8c9f88d36fb0a8dbcbee88b8d1ff6d44676 not found: ID does not exist"
Sep 07 22:49:35.265488 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.265461    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"]
Sep 07 22:49:35.267488 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:35.267469    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-52b103/explicit-model-52b103-kserve-6745b45859-84nq7"]
Sep 07 22:49:36.090431 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:36.090394    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a4abc76-8b39-47f7-b6a2-50b701cc1924" path="/var/lib/kubelet/pods/2a4abc76-8b39-47f7-b6a2-50b701cc1924/volumes"
Sep 07 22:49:37.179870 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.179836    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"]
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180214    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a80db81d-425f-4884-bffc-a517f8d9f795" containerName="maas-api"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180227    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a80db81d-425f-4884-bffc-a517f8d9f795" containerName="maas-api"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180239    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a4abc76-8b39-47f7-b6a2-50b701cc1924" containerName="main"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180245    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a4abc76-8b39-47f7-b6a2-50b701cc1924" containerName="main"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180269    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="20c7ec13-39d6-4971-8248-fbf77383762a" containerName="payload-processing"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180276    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="20c7ec13-39d6-4971-8248-fbf77383762a" containerName="payload-processing"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180359    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a80db81d-425f-4884-bffc-a517f8d9f795" containerName="maas-api"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180369    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="20c7ec13-39d6-4971-8248-fbf77383762a" containerName="payload-processing"
Sep 07 22:49:37.180477 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.180375    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a4abc76-8b39-47f7-b6a2-50b701cc1924" containerName="main"
Sep 07 22:49:37.184516 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.184494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.186940 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.186919    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-e19e20-openshift-default-dockercfg-vftfl\""
Sep 07 22:49:37.196638 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.196610    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"]
Sep 07 22:49:37.262501 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262460    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-credential-socket\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262508    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrhl\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-kube-api-access-tvrhl\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262604    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-token\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262634    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istiod-ca-cert\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262691 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262656    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-podinfo\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262885 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262720    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-certs\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262885 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262754    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-envoy\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262885 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262782    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-data\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.262885 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.262799    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-socket\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-credential-socket\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364084    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrhl\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-kube-api-access-tvrhl\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364134    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-token\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364156    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istiod-ca-cert\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364172    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-podinfo\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364200    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-certs\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-envoy\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364253    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-data\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364672 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364278    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-socket\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364672 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364449    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-credential-socket\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364672 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-certs\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364780 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364744    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-socket\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.364834 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.364774    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-data\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.365169 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.365144    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istiod-ca-cert\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.366634 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.366610    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-envoy\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.366758 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.366739    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-podinfo\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.371874 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.371852    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-token\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.371959 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.371905    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrhl\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-kube-api-access-tvrhl\") pod \"e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") " pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.497690 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.497599    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:37.772045 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.771955    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"]
Sep 07 22:49:37.775137 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:49:37.775109    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69bccca5_0a13_4252_99ce_bca3a6d9bc42.slice/crio-8f280e526f414f68628b0cc4ab9540cfdbac737c19791dece914a71eb9e6ae9f WatchSource:0}: Error finding container 8f280e526f414f68628b0cc4ab9540cfdbac737c19791dece914a71eb9e6ae9f: Status 404 returned error can't find the container with id 8f280e526f414f68628b0cc4ab9540cfdbac737c19791dece914a71eb9e6ae9f
Sep 07 22:49:37.777460 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.777429    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:49:37.777529 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.777506    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:49:37.777569 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:37.777556    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:49:38.263588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:38.263546    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd" event={"ID":"69bccca5-0a13-4252-99ce-bca3a6d9bc42","Type":"ContainerStarted","Data":"4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692"}
Sep 07 22:49:38.264015 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:38.263607    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd" event={"ID":"69bccca5-0a13-4252-99ce-bca3a6d9bc42","Type":"ContainerStarted","Data":"8f280e526f414f68628b0cc4ab9540cfdbac737c19791dece914a71eb9e6ae9f"}
Sep 07 22:49:38.284615 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:38.284555    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd" podStartSLOduration=1.28453979 podStartE2EDuration="1.28453979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:38.281715891 +0000 UTC m=+1044.747872886" watchObservedRunningTime="2026-09-07 22:49:38.28453979 +0000 UTC m=+1044.750696781"
Sep 07 22:49:38.498817 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:38.498774    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:38.504731 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:38.504702    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:39.269071 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:39.269026    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:39.270324 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:39.270301    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:49:41.224206 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:41.224167    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:49:53.459580 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.459537    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"]
Sep 07 22:49:53.463379 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.463359    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.466978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.466945    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e19e20\"/\"default-dockercfg-gxrhz\""
Sep 07 22:49:53.467139 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.466943    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e19e20\"/\"kube-root-ca.crt\""
Sep 07 22:49:53.467139 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.467001    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-e19e20\"/\"noten-model-e19e20-kserve-self-signed-certs\""
Sep 07 22:49:53.467322 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.467303    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-e19e20\"/\"openshift-service-ca.crt\""
Sep 07 22:49:53.470087 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.470065    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"]
Sep 07 22:49:53.520463 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.520427    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-dshm\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.520463 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.520472    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb87g\" (UniqueName: \"kubernetes.io/projected/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-kube-api-access-qb87g\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.520705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.520561    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tmp-dir\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.520705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.520598    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-model-cache\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.520705 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.520659    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-home\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.520828 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.520714    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tls-certs\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.621751 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.621709    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-home\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.621751 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.621753    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tls-certs\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.622013 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.621821    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-dshm\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.622013 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.621879    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb87g\" (UniqueName: \"kubernetes.io/projected/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-kube-api-access-qb87g\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.622013 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.621946    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tmp-dir\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.622013 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.621984    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-model-cache\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.622374 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.622350    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-home\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.622466 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.622399    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-model-cache\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.622466 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.622423    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tmp-dir\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.624645 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.624620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-dshm\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.624939 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.624919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tls-certs\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.630757 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.630736    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb87g\" (UniqueName: \"kubernetes.io/projected/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-kube-api-access-qb87g\") pod \"noten-model-e19e20-kserve-57bc97ccf9-dv8v8\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") " pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:53.775729 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:53.775645    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:49:54.052985 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:54.052960    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"]
Sep 07 22:49:54.054592 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:49:54.054561    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda69d85fe_1bc3_4a68_ab52_c1696a2e52dc.slice/crio-22b0c692b9a56bd81931ede2faec7d0efcf3fd5ce046a28a931b3c0778b63755 WatchSource:0}: Error finding container 22b0c692b9a56bd81931ede2faec7d0efcf3fd5ce046a28a931b3c0778b63755: Status 404 returned error can't find the container with id 22b0c692b9a56bd81931ede2faec7d0efcf3fd5ce046a28a931b3c0778b63755
Sep 07 22:49:54.357924 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:54.357887    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8" event={"ID":"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc","Type":"ContainerStarted","Data":"0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a"}
Sep 07 22:49:54.357924 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:54.357931    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8" event={"ID":"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc","Type":"ContainerStarted","Data":"22b0c692b9a56bd81931ede2faec7d0efcf3fd5ce046a28a931b3c0778b63755"}
Sep 07 22:49:54.375414 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:54.375367    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8" podStartSLOduration=1.375351478 podStartE2EDuration="1.375351478s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:49:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:49:54.372790949 +0000 UTC m=+1060.838947944" watchObservedRunningTime="2026-09-07 22:49:54.375351478 +0000 UTC m=+1060.841508473"
Sep 07 22:49:55.974963 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:55.974928    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"]
Sep 07 22:49:56.090316 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:56.090253    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"]
Sep 07 22:49:56.090516 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:49:56.090484    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd" podUID="69bccca5-0a13-4252-99ce-bca3a6d9bc42" containerName="istio-proxy" containerID="cri-o://4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692" gracePeriod=30
Sep 07 22:50:01.350882 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.350809    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:50:01.391061 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391018    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-socket\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.391246 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391163    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-credential-socket\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.391361 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391273    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-data\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.391361 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391348    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-podinfo\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.391482 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391396    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvrhl\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-kube-api-access-tvrhl\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.391482 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391401    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:01.391482 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391428    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-envoy\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.392153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391666    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istiod-ca-cert\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.392153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391736    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-data" (OuterVolumeSpecName: "istio-data") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:01.392153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391783    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-token\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.392153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391822    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:01.392153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.391840    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-certs\") pod \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\" (UID: \"69bccca5-0a13-4252-99ce-bca3a6d9bc42\") "
Sep 07 22:50:01.392153 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.392055    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:01.392522 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.392218    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:01.392522 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.392318    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.392522 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.392336    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-credential-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.392522 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.392349    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-data\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.392522 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.392364    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istiod-ca-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.394167 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.394139    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:50:01.394468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.394422    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:01.394468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.394431    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-token" (OuterVolumeSpecName: "istio-token") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:01.394468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.394443    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-kube-api-access-tvrhl" (OuterVolumeSpecName: "kube-api-access-tvrhl") pod "69bccca5-0a13-4252-99ce-bca3a6d9bc42" (UID: "69bccca5-0a13-4252-99ce-bca3a6d9bc42"). InnerVolumeSpecName "kube-api-access-tvrhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:01.406179 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.406139    2582 generic.go:356] "Generic (PLEG): container finished" podID="69bccca5-0a13-4252-99ce-bca3a6d9bc42" containerID="4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692" exitCode=0
Sep 07 22:50:01.406340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.406200    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"
Sep 07 22:50:01.406340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.406214    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd" event={"ID":"69bccca5-0a13-4252-99ce-bca3a6d9bc42","Type":"ContainerDied","Data":"4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692"}
Sep 07 22:50:01.406340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.406269    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd" event={"ID":"69bccca5-0a13-4252-99ce-bca3a6d9bc42","Type":"ContainerDied","Data":"8f280e526f414f68628b0cc4ab9540cfdbac737c19791dece914a71eb9e6ae9f"}
Sep 07 22:50:01.406340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.406310    2582 scope.go:117] "RemoveContainer" containerID="4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692"
Sep 07 22:50:01.416597 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.416566    2582 scope.go:117] "RemoveContainer" containerID="4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692"
Sep 07 22:50:01.416932 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:01.416904    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692\": container with ID starting with 4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692 not found: ID does not exist" containerID="4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692"
Sep 07 22:50:01.417022 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.416946    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692"} err="failed to get container status \"4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692\": rpc error: code = NotFound desc = could not find container \"4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692\": container with ID starting with 4b41aa2678b889bc2f21a3ee51b74036b39ac966684a15602f540a3228ae5692 not found: ID does not exist"
Sep 07 22:50:01.431547 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.431512    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"]
Sep 07 22:50:01.435403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.435376    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-e19e20-openshift-default-84fdcff6b4-wm7kd"]
Sep 07 22:50:01.493780 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.493748    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-podinfo\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.493780 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.493775    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvrhl\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-kube-api-access-tvrhl\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.493780 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.493785    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-envoy\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.494018 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.493794    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69bccca5-0a13-4252-99ce-bca3a6d9bc42-istio-token\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:01.494018 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:01.493811    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69bccca5-0a13-4252-99ce-bca3a6d9bc42-workload-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:02.090226 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:02.090190    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69bccca5-0a13-4252-99ce-bca3a6d9bc42" path="/var/lib/kubelet/pods/69bccca5-0a13-4252-99ce-bca3a6d9bc42/volumes"
Sep 07 22:50:03.776630 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:03.776598    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:50:09.723438 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:09.723342    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"]
Sep 07 22:50:09.724002 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:09.723607    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955" podUID="a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" containerName="payload-pre-processing" containerID="cri-o://d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97" gracePeriod=30
Sep 07 22:50:09.983440 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:09.983371    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:50:10.073469 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.073439    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-plugins-config-volume\") pod \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") "
Sep 07 22:50:10.073679 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.073497    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrjc2\" (UniqueName: \"kubernetes.io/projected/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-kube-api-access-jrjc2\") pod \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\" (UID: \"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8\") "
Sep 07 22:50:10.073825 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.073794    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" (UID: "a63bdbe3-0058-47a5-879a-1a15e5fd1fd8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:10.073901 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.073888    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.075748 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.075715    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-kube-api-access-jrjc2" (OuterVolumeSpecName: "kube-api-access-jrjc2") pod "a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" (UID: "a63bdbe3-0058-47a5-879a-1a15e5fd1fd8"). InnerVolumeSpecName "kube-api-access-jrjc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:10.175260 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.175220    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrjc2\" (UniqueName: \"kubernetes.io/projected/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8-kube-api-access-jrjc2\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:10.461663 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.461623    2582 generic.go:356] "Generic (PLEG): container finished" podID="a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" containerID="d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97" exitCode=0
Sep 07 22:50:10.461882 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.461683    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"
Sep 07 22:50:10.461882 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.461709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955" event={"ID":"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8","Type":"ContainerDied","Data":"d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97"}
Sep 07 22:50:10.461882 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.461793    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955" event={"ID":"a63bdbe3-0058-47a5-879a-1a15e5fd1fd8","Type":"ContainerDied","Data":"c1cfb250c17b65feb62c8ee306ab5fbdd34a567fe5508ea94928e6a885a0f897"}
Sep 07 22:50:10.461882 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.461824    2582 scope.go:117] "RemoveContainer" containerID="d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97"
Sep 07 22:50:10.472883 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.472863    2582 scope.go:117] "RemoveContainer" containerID="d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97"
Sep 07 22:50:10.473435 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:10.473413    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97\": container with ID starting with d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97 not found: ID does not exist" containerID="d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97"
Sep 07 22:50:10.473533 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.473441    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97"} err="failed to get container status \"d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97\": rpc error: code = NotFound desc = could not find container \"d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97\": container with ID starting with d557bbf04add8580de032c12b125b40a19d2ca69b7f6f0917deee8b90fe3ea97 not found: ID does not exist"
Sep 07 22:50:10.478212 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.478188    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"]
Sep 07 22:50:10.481074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:10.481054    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-2872fd39-5fb7d5b7c8-58955"]
Sep 07 22:50:11.390180 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.390120    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8" podUID="a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" containerName="main" containerID="cri-o://0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a" gracePeriod=45
Sep 07 22:50:11.650778 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.650715    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:50:11.788852 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.788822    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tmp-dir\") pod \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") "
Sep 07 22:50:11.789053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.788862    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qb87g\" (UniqueName: \"kubernetes.io/projected/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-kube-api-access-qb87g\") pod \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") "
Sep 07 22:50:11.789053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.788891    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-home\") pod \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") "
Sep 07 22:50:11.789053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.788976    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-model-cache\") pod \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") "
Sep 07 22:50:11.789053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789042    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tls-certs\") pod \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") "
Sep 07 22:50:11.789320 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-dshm\") pod \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\" (UID: \"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc\") "
Sep 07 22:50:11.789320 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789141    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" (UID: "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:11.789320 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789190    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-home" (OuterVolumeSpecName: "home") pod "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" (UID: "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:11.789320 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789271    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-model-cache" (OuterVolumeSpecName: "model-cache") pod "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" (UID: "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:11.789587 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789566    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-model-cache\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:11.789647 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789618    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tmp-dir\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:11.789706 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.789653    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-home\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:11.791267 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.791241    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-dshm" (OuterVolumeSpecName: "dshm") pod "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" (UID: "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:11.791409 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.791368    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-kube-api-access-qb87g" (OuterVolumeSpecName: "kube-api-access-qb87g") pod "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" (UID: "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc"). InnerVolumeSpecName "kube-api-access-qb87g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:11.791498 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.791477    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" (UID: "a69d85fe-1bc3-4a68-ab52-c1696a2e52dc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:50:11.890352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.890313    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qb87g\" (UniqueName: \"kubernetes.io/projected/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-kube-api-access-qb87g\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:11.890352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.890348    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-tls-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:11.890352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:11.890360    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc-dshm\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:12.090653 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.090614    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" path="/var/lib/kubelet/pods/a63bdbe3-0058-47a5-879a-1a15e5fd1fd8/volumes"
Sep 07 22:50:12.477512 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.477427    2582 generic.go:356] "Generic (PLEG): container finished" podID="a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" containerID="0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a" exitCode=0
Sep 07 22:50:12.477512 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.477480    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"
Sep 07 22:50:12.477512 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.477488    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8" event={"ID":"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc","Type":"ContainerDied","Data":"0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a"}
Sep 07 22:50:12.478074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.477530    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8" event={"ID":"a69d85fe-1bc3-4a68-ab52-c1696a2e52dc","Type":"ContainerDied","Data":"22b0c692b9a56bd81931ede2faec7d0efcf3fd5ce046a28a931b3c0778b63755"}
Sep 07 22:50:12.478074 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.477556    2582 scope.go:117] "RemoveContainer" containerID="0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a"
Sep 07 22:50:12.489473 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.489441    2582 scope.go:117] "RemoveContainer" containerID="0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a"
Sep 07 22:50:12.489993 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:12.489974    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a\": container with ID starting with 0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a not found: ID does not exist" containerID="0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a"
Sep 07 22:50:12.490080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.490001    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a"} err="failed to get container status \"0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a\": rpc error: code = NotFound desc = could not find container \"0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a\": container with ID starting with 0af8a0d0426cc750581563f6af3ec4be5db44bb9e28f28ddf4470ff60c4f720a not found: ID does not exist"
Sep 07 22:50:12.496830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.496803    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"]
Sep 07 22:50:12.498876 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.498852    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-e19e20/noten-model-e19e20-kserve-57bc97ccf9-dv8v8"]
Sep 07 22:50:12.608784 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.608738    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"]
Sep 07 22:50:12.609148 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609132    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" containerName="payload-pre-processing"
Sep 07 22:50:12.609148 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609148    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" containerName="payload-pre-processing"
Sep 07 22:50:12.609252 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609169    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" containerName="main"
Sep 07 22:50:12.609252 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609175    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" containerName="main"
Sep 07 22:50:12.609252 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609190    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69bccca5-0a13-4252-99ce-bca3a6d9bc42" containerName="istio-proxy"
Sep 07 22:50:12.609252 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609196    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="69bccca5-0a13-4252-99ce-bca3a6d9bc42" containerName="istio-proxy"
Sep 07 22:50:12.609404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609260    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" containerName="main"
Sep 07 22:50:12.609404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609269    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="69bccca5-0a13-4252-99ce-bca3a6d9bc42" containerName="istio-proxy"
Sep 07 22:50:12.609404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.609278    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="a63bdbe3-0058-47a5-879a-1a15e5fd1fd8" containerName="payload-pre-processing"
Sep 07 22:50:12.613725 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.613700    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"
Sep 07 22:50:12.616722 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.616137    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 22:50:12.616722 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.616479    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:50:12.619217 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.619153    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"]
Sep 07 22:50:12.697403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.697361    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79245e47-d00e-40b1-b37b-c9a6300ea7f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z\" (UID: \"79245e47-d00e-40b1-b37b-c9a6300ea7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"
Sep 07 22:50:12.798961 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.798852    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79245e47-d00e-40b1-b37b-c9a6300ea7f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z\" (UID: \"79245e47-d00e-40b1-b37b-c9a6300ea7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"
Sep 07 22:50:12.799582 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.799557    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79245e47-d00e-40b1-b37b-c9a6300ea7f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z\" (UID: \"79245e47-d00e-40b1-b37b-c9a6300ea7f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"
Sep 07 22:50:12.925985 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:12.925948    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"
Sep 07 22:50:13.197443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:13.197415    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"]
Sep 07 22:50:13.199069 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:50:13.199028    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-6d4275457e9eed08430f46fca664c52d8978ae958e428e75426aa463fa766f0f WatchSource:0}: Error finding container 6d4275457e9eed08430f46fca664c52d8978ae958e428e75426aa463fa766f0f: Status 404 returned error can't find the container with id 6d4275457e9eed08430f46fca664c52d8978ae958e428e75426aa463fa766f0f
Sep 07 22:50:13.484312 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:13.484192    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z" event={"ID":"79245e47-d00e-40b1-b37b-c9a6300ea7f5","Type":"ContainerStarted","Data":"6d4275457e9eed08430f46fca664c52d8978ae958e428e75426aa463fa766f0f"}
Sep 07 22:50:14.094788 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:14.094703    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a69d85fe-1bc3-4a68-ab52-c1696a2e52dc" path="/var/lib/kubelet/pods/a69d85fe-1bc3-4a68-ab52-c1696a2e52dc/volumes"
Sep 07 22:50:14.492474 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:14.492379    2582 generic.go:356] "Generic (PLEG): container finished" podID="79245e47-d00e-40b1-b37b-c9a6300ea7f5" containerID="abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec" exitCode=0
Sep 07 22:50:14.492474 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:14.492459    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z" event={"ID":"79245e47-d00e-40b1-b37b-c9a6300ea7f5","Type":"ContainerDied","Data":"abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec"}
Sep 07 22:50:15.639075 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:15.639049    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"
Sep 07 22:50:15.725166 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:15.725130    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79245e47-d00e-40b1-b37b-c9a6300ea7f5-maas-api-service-ca\") pod \"79245e47-d00e-40b1-b37b-c9a6300ea7f5\" (UID: \"79245e47-d00e-40b1-b37b-c9a6300ea7f5\") "
Sep 07 22:50:15.725538 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:15.725516    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79245e47-d00e-40b1-b37b-c9a6300ea7f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "79245e47-d00e-40b1-b37b-c9a6300ea7f5" (UID: "79245e47-d00e-40b1-b37b-c9a6300ea7f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:15.825880 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:15.825837    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/79245e47-d00e-40b1-b37b-c9a6300ea7f5-maas-api-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:15.831905 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:15.831875    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:16.433530 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:16.433451    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:16.504253 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:16.504224    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"
Sep 07 22:50:16.504443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:16.504256    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z" event={"ID":"79245e47-d00e-40b1-b37b-c9a6300ea7f5","Type":"ContainerDied","Data":"6d4275457e9eed08430f46fca664c52d8978ae958e428e75426aa463fa766f0f"}
Sep 07 22:50:16.504443 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:16.504340    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d4275457e9eed08430f46fca664c52d8978ae958e428e75426aa463fa766f0f"
Sep 07 22:50:23.160189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.160153    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"]
Sep 07 22:50:23.160634 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.160397    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh" podUID="df3bb18c-95db-4e97-bfcf-c95641ba8a6c" containerName="payload-processing" containerID="cri-o://b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87" gracePeriod=30
Sep 07 22:50:23.168744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.168715    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"]
Sep 07 22:50:23.168962 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.168934    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8" podUID="833778f4-e80f-4ef0-baa5-2159e6175668" containerName="payload-pre-processing" containerID="cri-o://eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc" gracePeriod=30
Sep 07 22:50:23.340454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.340397    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"]
Sep 07 22:50:23.343933 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.343902    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-51cb36-l7n9z"]
Sep 07 22:50:23.445436 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.445412    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:50:23.448231 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.448211    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:50:23.546971 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.546937    2582 generic.go:356] "Generic (PLEG): container finished" podID="df3bb18c-95db-4e97-bfcf-c95641ba8a6c" containerID="b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87" exitCode=0
Sep 07 22:50:23.547187 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.547017    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"
Sep 07 22:50:23.547187 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.547028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh" event={"ID":"df3bb18c-95db-4e97-bfcf-c95641ba8a6c","Type":"ContainerDied","Data":"b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87"}
Sep 07 22:50:23.547187 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.547098    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh" event={"ID":"df3bb18c-95db-4e97-bfcf-c95641ba8a6c","Type":"ContainerDied","Data":"128e0b4117fbe8753fcbf09c29057150a5cb16fde9a10fca76c1c6b04ef20cd0"}
Sep 07 22:50:23.547187 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.547135    2582 scope.go:117] "RemoveContainer" containerID="b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87"
Sep 07 22:50:23.548475 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.548452    2582 generic.go:356] "Generic (PLEG): container finished" podID="833778f4-e80f-4ef0-baa5-2159e6175668" containerID="eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc" exitCode=0
Sep 07 22:50:23.548564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.548488    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"
Sep 07 22:50:23.548564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.548519    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8" event={"ID":"833778f4-e80f-4ef0-baa5-2159e6175668","Type":"ContainerDied","Data":"eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc"}
Sep 07 22:50:23.548564 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.548558    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8" event={"ID":"833778f4-e80f-4ef0-baa5-2159e6175668","Type":"ContainerDied","Data":"7cf3e419b8e9f80ceb23723acf4893ab5a90e5cb95b6ebf50168ff74260ab6b0"}
Sep 07 22:50:23.557435 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.557417    2582 scope.go:117] "RemoveContainer" containerID="b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87"
Sep 07 22:50:23.557740 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:23.557717    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87\": container with ID starting with b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87 not found: ID does not exist" containerID="b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87"
Sep 07 22:50:23.557826 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.557749    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87"} err="failed to get container status \"b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87\": rpc error: code = NotFound desc = could not find container \"b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87\": container with ID starting with b77d5e1a78a219c36b2959e23a5e4b9f2c232a76f2453c031cb3fd7db412ae87 not found: ID does not exist"
Sep 07 22:50:23.557826 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.557767    2582 scope.go:117] "RemoveContainer" containerID="eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc"
Sep 07 22:50:23.566297 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.566262    2582 scope.go:117] "RemoveContainer" containerID="eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc"
Sep 07 22:50:23.566633 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:23.566614    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc\": container with ID starting with eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc not found: ID does not exist" containerID="eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc"
Sep 07 22:50:23.566678 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.566643    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc"} err="failed to get container status \"eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc\": rpc error: code = NotFound desc = could not find container \"eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc\": container with ID starting with eca54a1c0f4ab797e49c8c5a9c3a1365b088641a484cc17f36b32e7244367bdc not found: ID does not exist"
Sep 07 22:50:23.604125 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.604091    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x8w6p\" (UniqueName: \"kubernetes.io/projected/833778f4-e80f-4ef0-baa5-2159e6175668-kube-api-access-x8w6p\") pod \"833778f4-e80f-4ef0-baa5-2159e6175668\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") "
Sep 07 22:50:23.604274 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.604133    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-plugins-config-volume\") pod \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") "
Sep 07 22:50:23.604274 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.604169    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833778f4-e80f-4ef0-baa5-2159e6175668-plugins-config-volume\") pod \"833778f4-e80f-4ef0-baa5-2159e6175668\" (UID: \"833778f4-e80f-4ef0-baa5-2159e6175668\") "
Sep 07 22:50:23.604274 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.604237    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjz8k\" (UniqueName: \"kubernetes.io/projected/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-kube-api-access-pjz8k\") pod \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\" (UID: \"df3bb18c-95db-4e97-bfcf-c95641ba8a6c\") "
Sep 07 22:50:23.604588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.604553    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/833778f4-e80f-4ef0-baa5-2159e6175668-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "833778f4-e80f-4ef0-baa5-2159e6175668" (UID: "833778f4-e80f-4ef0-baa5-2159e6175668"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:23.604702 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.604590    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df3bb18c-95db-4e97-bfcf-c95641ba8a6c" (UID: "df3bb18c-95db-4e97-bfcf-c95641ba8a6c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:23.606360 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.606337    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-kube-api-access-pjz8k" (OuterVolumeSpecName: "kube-api-access-pjz8k") pod "df3bb18c-95db-4e97-bfcf-c95641ba8a6c" (UID: "df3bb18c-95db-4e97-bfcf-c95641ba8a6c"). InnerVolumeSpecName "kube-api-access-pjz8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:23.606455 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.606336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/833778f4-e80f-4ef0-baa5-2159e6175668-kube-api-access-x8w6p" (OuterVolumeSpecName: "kube-api-access-x8w6p") pod "833778f4-e80f-4ef0-baa5-2159e6175668" (UID: "833778f4-e80f-4ef0-baa5-2159e6175668"). InnerVolumeSpecName "kube-api-access-x8w6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:23.705838 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.705741    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjz8k\" (UniqueName: \"kubernetes.io/projected/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-kube-api-access-pjz8k\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:23.705838 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.705774    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x8w6p\" (UniqueName: \"kubernetes.io/projected/833778f4-e80f-4ef0-baa5-2159e6175668-kube-api-access-x8w6p\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:23.705838 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.705791    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df3bb18c-95db-4e97-bfcf-c95641ba8a6c-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:23.705838 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.705806    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/833778f4-e80f-4ef0-baa5-2159e6175668-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:23.871167 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.871136    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"]
Sep 07 22:50:23.875053 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.875025    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-51cb36-95fd545b6-sxss8"]
Sep 07 22:50:23.884106 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.884081    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"]
Sep 07 22:50:23.887999 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:23.887975    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-51cb36-75bf547465-9vwrh"]
Sep 07 22:50:24.090189 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:24.090155    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79245e47-d00e-40b1-b37b-c9a6300ea7f5" path="/var/lib/kubelet/pods/79245e47-d00e-40b1-b37b-c9a6300ea7f5/volumes"
Sep 07 22:50:24.090597 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:24.090583    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="833778f4-e80f-4ef0-baa5-2159e6175668" path="/var/lib/kubelet/pods/833778f4-e80f-4ef0-baa5-2159e6175668/volumes"
Sep 07 22:50:24.090907 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:24.090895    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df3bb18c-95db-4e97-bfcf-c95641ba8a6c" path="/var/lib/kubelet/pods/df3bb18c-95db-4e97-bfcf-c95641ba8a6c/volumes"
Sep 07 22:50:26.475410 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:26.475378    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:29.586601 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.586561    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"]
Sep 07 22:50:29.587080 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587063    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79245e47-d00e-40b1-b37b-c9a6300ea7f5" containerName="revoke-keys"
Sep 07 22:50:29.587125 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587083    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="79245e47-d00e-40b1-b37b-c9a6300ea7f5" containerName="revoke-keys"
Sep 07 22:50:29.587125 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587103    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df3bb18c-95db-4e97-bfcf-c95641ba8a6c" containerName="payload-processing"
Sep 07 22:50:29.587125 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587113    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="df3bb18c-95db-4e97-bfcf-c95641ba8a6c" containerName="payload-processing"
Sep 07 22:50:29.587215 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587133    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="833778f4-e80f-4ef0-baa5-2159e6175668" containerName="payload-pre-processing"
Sep 07 22:50:29.587215 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587143    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="833778f4-e80f-4ef0-baa5-2159e6175668" containerName="payload-pre-processing"
Sep 07 22:50:29.587299 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587223    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="79245e47-d00e-40b1-b37b-c9a6300ea7f5" containerName="revoke-keys"
Sep 07 22:50:29.587299 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587237    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="833778f4-e80f-4ef0-baa5-2159e6175668" containerName="payload-pre-processing"
Sep 07 22:50:29.587299 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.587246    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="df3bb18c-95db-4e97-bfcf-c95641ba8a6c" containerName="payload-processing"
Sep 07 22:50:29.592223 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.592200    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.594606 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.594584    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-82409b-openshift-default-dockercfg-z4rls\""
Sep 07 22:50:29.603683 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.603658    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"]
Sep 07 22:50:29.764237 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764190    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-credential-socket\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764452 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764246    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c26ae9-651d-471b-b939-6805f3c5f832-istiod-ca-cert\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764452 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764336    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-socket\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764452 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764420    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-token\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764452 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764448    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-envoy\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764667 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764474    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-certs\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764667 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764511    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-podinfo\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764667 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764541    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-data\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.764667 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.764578    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7w257\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-kube-api-access-7w257\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866023 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.865930    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-envoy\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866023 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.865980    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-certs\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866023 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866014    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-podinfo\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866120    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-data\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7w257\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-kube-api-access-7w257\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866183    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-credential-socket\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866224    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c26ae9-651d-471b-b939-6805f3c5f832-istiod-ca-cert\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866248    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-socket\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866628 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866366    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-token\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866628 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866553    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-data\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866628 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866605    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-credential-socket\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866787 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866765    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-socket\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.866973 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.866950    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-certs\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.867075 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.867013    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c26ae9-651d-471b-b939-6805f3c5f832-istiod-ca-cert\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.868377 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.868352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-envoy\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.868694 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.868671    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-podinfo\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.874997 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.874969    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-token\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.875113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.875097    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7w257\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-kube-api-access-7w257\") pod \"e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") " pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.904361 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.904311    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:29.995786 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.995731    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"]
Sep 07 22:50:29.996037 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:29.996007    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy" containerID="cri-o://5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0" gracePeriod=30
Sep 07 22:50:30.179944 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.179913    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"]
Sep 07 22:50:30.181890 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:50:30.181861    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0c26ae9_651d_471b_b939_6805f3c5f832.slice/crio-1d2000760c959bce1e4f23044944580b72dd4f7d91283fec96928f16d7833b24 WatchSource:0}: Error finding container 1d2000760c959bce1e4f23044944580b72dd4f7d91283fec96928f16d7833b24: Status 404 returned error can't find the container with id 1d2000760c959bce1e4f23044944580b72dd4f7d91283fec96928f16d7833b24
Sep 07 22:50:30.184507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.184474    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:50:30.184578 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.184540    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:50:30.184624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.184583    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:50:30.591578 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.591540    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" event={"ID":"f0c26ae9-651d-471b-b939-6805f3c5f832","Type":"ContainerStarted","Data":"c61aade99215aa511d4013705416bd6175d7710207989982f32df33e87e0233f"}
Sep 07 22:50:30.591986 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.591630    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" event={"ID":"f0c26ae9-651d-471b-b939-6805f3c5f832","Type":"ContainerStarted","Data":"1d2000760c959bce1e4f23044944580b72dd4f7d91283fec96928f16d7833b24"}
Sep 07 22:50:30.610482 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.610420    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podStartSLOduration=1.610401515 podStartE2EDuration="1.610401515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:30.608655336 +0000 UTC m=+1097.074812333" watchObservedRunningTime="2026-09-07 22:50:30.610401515 +0000 UTC m=+1097.076558510"
Sep 07 22:50:30.800131 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:30.800098    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:30.905388 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:30.905261    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:31.910317 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:31.910272    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:32.602893 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:32.602852    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:32.603872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:32.603849    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:34.313589 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.313554    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"]
Sep 07 22:50:34.317700 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.317675    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.320178 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.320155    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-82409b\""
Sep 07 22:50:34.320313 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.320160    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-82409b-dockercfg-fl92x\""
Sep 07 22:50:34.326630 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.326605    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"]
Sep 07 22:50:34.409556 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.409517    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d50b72b-1203-43b7-b38d-9170e3e6c824-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.409746 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.409574    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trsn2\" (UniqueName: \"kubernetes.io/projected/7d50b72b-1203-43b7-b38d-9170e3e6c824-kube-api-access-trsn2\") pod \"payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.510099 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.510065    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d50b72b-1203-43b7-b38d-9170e3e6c824-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.510328 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.510121    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trsn2\" (UniqueName: \"kubernetes.io/projected/7d50b72b-1203-43b7-b38d-9170e3e6c824-kube-api-access-trsn2\") pod \"payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.510779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.510758    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d50b72b-1203-43b7-b38d-9170e3e6c824-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.518317 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.518267    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trsn2\" (UniqueName: \"kubernetes.io/projected/7d50b72b-1203-43b7-b38d-9170e3e6c824-kube-api-access-trsn2\") pod \"payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.630460 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.630377    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:34.904000 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:34.903803    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"]
Sep 07 22:50:35.238054 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.238027    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:50:35.315128 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315089    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e2d636-9b36-4de1-b874-17954e4ec581-istiod-ca-cert\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315635 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315147    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tm88x\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-kube-api-access-tm88x\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315635 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315190    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-envoy\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315635 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315224    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-socket\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315655    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:35.315793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315569    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e2d636-9b36-4de1-b874-17954e4ec581-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:35.315793 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315739    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-istio-token\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315809    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-credential-socket\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315840    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-certs\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315881    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-data\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.315954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.315940    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e2d636-9b36-4de1-b874-17954e4ec581-istio-podinfo\") pod \"38e2d636-9b36-4de1-b874-17954e4ec581\" (UID: \"38e2d636-9b36-4de1-b874-17954e4ec581\") "
Sep 07 22:50:35.316642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.316252    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38e2d636-9b36-4de1-b874-17954e4ec581-istiod-ca-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.316642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.316275    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.316642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.316325    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:35.316642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.316545    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:35.316642 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.316616    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-data" (OuterVolumeSpecName: "istio-data") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:35.317953 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.317928    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:50:35.318806 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.318764    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-istio-token" (OuterVolumeSpecName: "istio-token") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:35.318902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.318865    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-kube-api-access-tm88x" (OuterVolumeSpecName: "kube-api-access-tm88x") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "kube-api-access-tm88x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:35.318972 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.318891    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38e2d636-9b36-4de1-b874-17954e4ec581-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "38e2d636-9b36-4de1-b874-17954e4ec581" (UID: "38e2d636-9b36-4de1-b874-17954e4ec581"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:50:35.417207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.417106    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-istio-token\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.417207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.417148    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-credential-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.417207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.417163    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-workload-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.417207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.417177    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-data\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.417207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.417189    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38e2d636-9b36-4de1-b874-17954e4ec581-istio-podinfo\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.417207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.417202    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm88x\" (UniqueName: \"kubernetes.io/projected/38e2d636-9b36-4de1-b874-17954e4ec581-kube-api-access-tm88x\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.417207 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.417215    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38e2d636-9b36-4de1-b874-17954e4ec581-istio-envoy\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:35.623505 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.623467    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht" event={"ID":"7d50b72b-1203-43b7-b38d-9170e3e6c824","Type":"ContainerStarted","Data":"38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad"}
Sep 07 22:50:35.623505 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.623536    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:35.623505 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.623561    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht" event={"ID":"7d50b72b-1203-43b7-b38d-9170e3e6c824","Type":"ContainerStarted","Data":"d1821f880026b81b12774b6b357eededda4ad8ab84917e1cc4de77b571d29afe"}
Sep 07 22:50:35.625334 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.625276    2582 generic.go:356] "Generic (PLEG): container finished" podID="38e2d636-9b36-4de1-b874-17954e4ec581" containerID="5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0" exitCode=0
Sep 07 22:50:35.625334 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.625323    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" event={"ID":"38e2d636-9b36-4de1-b874-17954e4ec581","Type":"ContainerDied","Data":"5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0"}
Sep 07 22:50:35.625495 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.625348    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"
Sep 07 22:50:35.625495 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.625380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7" event={"ID":"38e2d636-9b36-4de1-b874-17954e4ec581","Type":"ContainerDied","Data":"ae7f9a5bf59f04a57ce692c9adae990e424a634e74ae99843d6baabdde6d1309"}
Sep 07 22:50:35.625495 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.625411    2582 scope.go:117] "RemoveContainer" containerID="5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0"
Sep 07 22:50:35.636656 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.636512    2582 scope.go:117] "RemoveContainer" containerID="36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9"
Sep 07 22:50:35.638904 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.638852    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht" podStartSLOduration=1.638830231 podStartE2EDuration="1.638830231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:35.637965917 +0000 UTC m=+1102.104122913" watchObservedRunningTime="2026-09-07 22:50:35.638830231 +0000 UTC m=+1102.104987229"
Sep 07 22:50:35.646480 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.646460    2582 scope.go:117] "RemoveContainer" containerID="5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0"
Sep 07 22:50:35.646800 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:35.646782    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0\": container with ID starting with 5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0 not found: ID does not exist" containerID="5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0"
Sep 07 22:50:35.646848 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.646813    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0"} err="failed to get container status \"5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0\": rpc error: code = NotFound desc = could not find container \"5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0\": container with ID starting with 5022fbdae34ac7ba35dbdff7359fe5713cc83436f4cdfeca7aaf17f565239ab0 not found: ID does not exist"
Sep 07 22:50:35.646848 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.646831    2582 scope.go:117] "RemoveContainer" containerID="36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9"
Sep 07 22:50:35.647107 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:35.647087    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9\": container with ID starting with 36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9 not found: ID does not exist" containerID="36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9"
Sep 07 22:50:35.647173 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.647111    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9"} err="failed to get container status \"36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9\": rpc error: code = NotFound desc = could not find container \"36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9\": container with ID starting with 36c4ecf151f32d9fe2881a14f9283be50b4f9731175420f31d9d5379f09279d9 not found: ID does not exist"
Sep 07 22:50:35.651172 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.651145    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"]
Sep 07 22:50:35.656954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:35.656932    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-51cb36-openshift-default-bfcc57f58-r8fm7"]
Sep 07 22:50:36.091591 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:36.091553    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" path="/var/lib/kubelet/pods/38e2d636-9b36-4de1-b874-17954e4ec581/volumes"
Sep 07 22:50:36.519711 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:36.519612    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:41.146323 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.140228    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"]
Sep 07 22:50:41.147249 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.147213    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5" podUID="03a20c25-28a4-4267-b7ec-a19fca4b55f2" containerName="payload-pre-processing" containerID="cri-o://d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564" gracePeriod=30
Sep 07 22:50:41.429309 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.424265    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:50:41.474553 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.474519    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jgmm\" (UniqueName: \"kubernetes.io/projected/03a20c25-28a4-4267-b7ec-a19fca4b55f2-kube-api-access-5jgmm\") pod \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") "
Sep 07 22:50:41.474740 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.474579    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a20c25-28a4-4267-b7ec-a19fca4b55f2-plugins-config-volume\") pod \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\" (UID: \"03a20c25-28a4-4267-b7ec-a19fca4b55f2\") "
Sep 07 22:50:41.474925 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.474901    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03a20c25-28a4-4267-b7ec-a19fca4b55f2-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "03a20c25-28a4-4267-b7ec-a19fca4b55f2" (UID: "03a20c25-28a4-4267-b7ec-a19fca4b55f2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:50:41.476789 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.476766    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03a20c25-28a4-4267-b7ec-a19fca4b55f2-kube-api-access-5jgmm" (OuterVolumeSpecName: "kube-api-access-5jgmm") pod "03a20c25-28a4-4267-b7ec-a19fca4b55f2" (UID: "03a20c25-28a4-4267-b7ec-a19fca4b55f2"). InnerVolumeSpecName "kube-api-access-5jgmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:50:41.576086 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.576050    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jgmm\" (UniqueName: \"kubernetes.io/projected/03a20c25-28a4-4267-b7ec-a19fca4b55f2-kube-api-access-5jgmm\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:41.576086 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.576082    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/03a20c25-28a4-4267-b7ec-a19fca4b55f2-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:50:41.660203 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.660113    2582 generic.go:356] "Generic (PLEG): container finished" podID="03a20c25-28a4-4267-b7ec-a19fca4b55f2" containerID="d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564" exitCode=0
Sep 07 22:50:41.660203 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.660167    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"
Sep 07 22:50:41.660414 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.660209    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5" event={"ID":"03a20c25-28a4-4267-b7ec-a19fca4b55f2","Type":"ContainerDied","Data":"d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564"}
Sep 07 22:50:41.660414 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.660264    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5" event={"ID":"03a20c25-28a4-4267-b7ec-a19fca4b55f2","Type":"ContainerDied","Data":"d98f80f412c456765b8cb671b5ae2b0d7cd3003a5ff59151049c5c2df7b9918a"}
Sep 07 22:50:41.660414 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.660309    2582 scope.go:117] "RemoveContainer" containerID="d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564"
Sep 07 22:50:41.669535 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.669519    2582 scope.go:117] "RemoveContainer" containerID="d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564"
Sep 07 22:50:41.669809 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:41.669788    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564\": container with ID starting with d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564 not found: ID does not exist" containerID="d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564"
Sep 07 22:50:41.669876 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.669816    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564"} err="failed to get container status \"d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564\": rpc error: code = NotFound desc = could not find container \"d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564\": container with ID starting with d79f02f59328102934f8736f3d252ac405ddb9839f31070e3e2080b408dae564 not found: ID does not exist"
Sep 07 22:50:41.681482 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.681454    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"]
Sep 07 22:50:41.686492 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:41.686468    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-79b9aa-7b644d78db-wc5r5"]
Sep 07 22:50:42.059824 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:42.059788    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:42.060020 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:42.059831    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:42.090617 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:42.090571    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03a20c25-28a4-4267-b7ec-a19fca4b55f2" path="/var/lib/kubelet/pods/03a20c25-28a4-4267-b7ec-a19fca4b55f2/volumes"
Sep 07 22:50:45.791467 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:45.791427    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:46.529297 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:46.529234    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:50:46.634487 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:46.634450    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:50:47.604340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.604258    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:50:47.604340 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.604325    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 07 22:50:47.694712 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.694622    2582 generic.go:356] "Generic (PLEG): container finished" podID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerID="c61aade99215aa511d4013705416bd6175d7710207989982f32df33e87e0233f" exitCode=0
Sep 07 22:50:47.694712 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.694693    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" event={"ID":"f0c26ae9-651d-471b-b939-6805f3c5f832","Type":"ContainerDied","Data":"c61aade99215aa511d4013705416bd6175d7710207989982f32df33e87e0233f"}
Sep 07 22:50:47.695125 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.695111    2582 scope.go:117] "RemoveContainer" containerID="c61aade99215aa511d4013705416bd6175d7710207989982f32df33e87e0233f"
Sep 07 22:50:47.696194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.696168    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:50:47.696268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.696248    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:50:47.696338 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:47.696306    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:50:48.701677 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:48.701643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" event={"ID":"f0c26ae9-651d-471b-b939-6805f3c5f832","Type":"ContainerStarted","Data":"9b961c334dd604573421140e5c7a3a4873f5123b24d5a2fe7cdc01788bac1266"}
Sep 07 22:50:48.905059 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:48.905023    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:48.905356 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:48.905334    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:50:48.905429 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:48.905384    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 07 22:50:49.905244 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:49.905197    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:50:49.905663 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:49.905314    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 07 22:50:50.905279 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:50.905238    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:50:50.905710 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:50.905333    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 07 22:50:51.905561 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:51.905523    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:50:51.905971 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:51.905583    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 07 22:50:52.734589 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.734510    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"]
Sep 07 22:50:52.735068 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735047    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03a20c25-28a4-4267-b7ec-a19fca4b55f2" containerName="payload-pre-processing"
Sep 07 22:50:52.735159 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735072    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="03a20c25-28a4-4267-b7ec-a19fca4b55f2" containerName="payload-pre-processing"
Sep 07 22:50:52.735159 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735094    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy"
Sep 07 22:50:52.735159 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735103    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy"
Sep 07 22:50:52.735159 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735118    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy"
Sep 07 22:50:52.735159 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735127    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy"
Sep 07 22:50:52.735444 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735248    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="03a20c25-28a4-4267-b7ec-a19fca4b55f2" containerName="payload-pre-processing"
Sep 07 22:50:52.735444 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.735262    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy"
Sep 07 22:50:52.742417 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.742376    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.745626 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.745588    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w3-1d0395-serving-cert\""
Sep 07 22:50:52.745821 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.745668    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 22:50:52.746031 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.745991    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 22:50:52.746545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.746509    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 22:50:52.746671 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.746561    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 22:50:52.749514 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.749486    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"]
Sep 07 22:50:52.787125 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.787090    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec052a6a-8e39-436b-9608-1ff4fa39dd77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.787322 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.787138    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.787322 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.787233    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.787405 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.787328    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/ec052a6a-8e39-436b-9608-1ff4fa39dd77-kube-api-access-npgpl\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.888420 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.888375    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.888620 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.888457    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.888620 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.888511    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/ec052a6a-8e39-436b-9608-1ff4fa39dd77-kube-api-access-npgpl\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.888620 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.888555    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec052a6a-8e39-436b-9608-1ff4fa39dd77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.888620 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:52.888594    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-serving-cert: secret "maas-api-e2e-shared-a-w3-1d0395-serving-cert" not found
Sep 07 22:50:52.888787 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:52.888663    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls podName:ec052a6a-8e39-436b-9608-1ff4fa39dd77 nodeName:}" failed. No retries permitted until 2026-09-07 22:50:53.388646124 +0000 UTC m=+1119.854803097 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls") pod "maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt" (UID: "ec052a6a-8e39-436b-9608-1ff4fa39dd77") : secret "maas-api-e2e-shared-a-w3-1d0395-serving-cert" not found
Sep 07 22:50:52.889278 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.889255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec052a6a-8e39-436b-9608-1ff4fa39dd77-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.890969 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.890942    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-metrics-certs\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.897966 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.897941    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/ec052a6a-8e39-436b-9608-1ff4fa39dd77-kube-api-access-npgpl\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:52.904892 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.904860    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused" start-of-body=
Sep 07 22:50:52.905020 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:52.904943    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.58:15021/healthz/ready\": dial tcp 10.134.0.58:15021: connect: connection refused"
Sep 07 22:50:53.392841 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.392801    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:53.395386 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.395358    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls\") pod \"maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:53.399130 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.399105    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"]
Sep 07 22:50:53.399539 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.399526    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e2d636-9b36-4de1-b874-17954e4ec581" containerName="istio-proxy"
Sep 07 22:50:53.401687 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.401669    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.404020 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.403997    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-1d0395\""
Sep 07 22:50:53.404255 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.404236    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-1d0395-dockercfg-tbh7w\""
Sep 07 22:50:53.416348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.413074    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"]
Sep 07 22:50:53.494137 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.494097    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg96m\" (UniqueName: \"kubernetes.io/projected/e1e88b33-cc4f-416c-9001-c5ea01420519-kube-api-access-sg96m\") pod \"payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.494338 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.494156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e88b33-cc4f-416c-9001-c5ea01420519-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.595554 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.595516    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg96m\" (UniqueName: \"kubernetes.io/projected/e1e88b33-cc4f-416c-9001-c5ea01420519-kube-api-access-sg96m\") pod \"payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.595764 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.595584    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e88b33-cc4f-416c-9001-c5ea01420519-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.596277 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.596255    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e88b33-cc4f-416c-9001-c5ea01420519-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.605115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.605075    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg96m\" (UniqueName: \"kubernetes.io/projected/e1e88b33-cc4f-416c-9001-c5ea01420519-kube-api-access-sg96m\") pod \"payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.656952 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.656836    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:53.718339 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.714256    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:53.910868 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.910773    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:53.911132 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.911107    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:53.911979 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.911959    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:50:53.930601 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.930422    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"]
Sep 07 22:50:53.932461 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:50:53.932425    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec052a6a_8e39_436b_9608_1ff4fa39dd77.slice/crio-4dc17dc7ab797ef1b5b33b497a0b96df8ca405030a5162fae7d7c749a0fe0887 WatchSource:0}: Error finding container 4dc17dc7ab797ef1b5b33b497a0b96df8ca405030a5162fae7d7c749a0fe0887: Status 404 returned error can't find the container with id 4dc17dc7ab797ef1b5b33b497a0b96df8ca405030a5162fae7d7c749a0fe0887
Sep 07 22:50:53.933906 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:53.933888    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:50:54.002780 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.002756    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"]
Sep 07 22:50:54.005356 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:50:54.005317    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1e88b33_cc4f_416c_9001_c5ea01420519.slice/crio-fb56ca17334aea4a9fc95f1a546be43c9dc88fecfdd8de1f36bd61713b7470a9 WatchSource:0}: Error finding container fb56ca17334aea4a9fc95f1a546be43c9dc88fecfdd8de1f36bd61713b7470a9: Status 404 returned error can't find the container with id fb56ca17334aea4a9fc95f1a546be43c9dc88fecfdd8de1f36bd61713b7470a9
Sep 07 22:50:54.744531 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.744497    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt" event={"ID":"ec052a6a-8e39-436b-9608-1ff4fa39dd77","Type":"ContainerStarted","Data":"077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5"}
Sep 07 22:50:54.744986 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.744553    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt" event={"ID":"ec052a6a-8e39-436b-9608-1ff4fa39dd77","Type":"ContainerStarted","Data":"4dc17dc7ab797ef1b5b33b497a0b96df8ca405030a5162fae7d7c749a0fe0887"}
Sep 07 22:50:54.744986 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.744683    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:50:54.746264 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.746237    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk" event={"ID":"e1e88b33-cc4f-416c-9001-c5ea01420519","Type":"ContainerStarted","Data":"935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f"}
Sep 07 22:50:54.746424 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.746291    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk" event={"ID":"e1e88b33-cc4f-416c-9001-c5ea01420519","Type":"ContainerStarted","Data":"fb56ca17334aea4a9fc95f1a546be43c9dc88fecfdd8de1f36bd61713b7470a9"}
Sep 07 22:50:54.746424 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.746335    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:50:54.762269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.762225    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt" podStartSLOduration=2.4608660430000002 podStartE2EDuration="2.762208435s" totalImagesPullingTime="301.342392ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:54.760698153 +0000 UTC m=+1121.226855142" watchObservedRunningTime="2026-09-07 22:50:54.762208435 +0000 UTC m=+1121.228365430"
Sep 07 22:50:54.777538 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:50:54.777482    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk" podStartSLOduration=1.7774671149999999 podStartE2EDuration="1.777467115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:50:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:50:54.775098461 +0000 UTC m=+1121.241255457" watchObservedRunningTime="2026-09-07 22:50:54.777467115 +0000 UTC m=+1121.243624154"
Sep 07 22:50:56.574513 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:50:56.574475    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:51:00.756918 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:00.756888    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 22:51:00.787883 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:51:00.787843    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:51:01.872978 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.872940    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"]
Sep 07 22:51:01.876406 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.876378    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:01.878717 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.878691    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-5483b1-serving-cert\""
Sep 07 22:51:01.887667 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.887638    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"]
Sep 07 22:51:01.976545 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.976503    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:01.976763 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.976568    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:01.976763 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.976594    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:01.976763 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:01.976619    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6w8t\" (UniqueName: \"kubernetes.io/projected/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-kube-api-access-z6w8t\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.077425 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.077382    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.077612 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.077436    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6w8t\" (UniqueName: \"kubernetes.io/projected/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-kube-api-access-z6w8t\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.077612 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.077519    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.077612 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.077566    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.078152 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.078117    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.080004 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.079981    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.080099 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.080012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.084766 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.084741    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6w8t\" (UniqueName: \"kubernetes.io/projected/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-kube-api-access-z6w8t\") pod \"maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.189474 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.189385    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:02.466668 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.466636    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"]
Sep 07 22:51:02.467532 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:51:02.467501    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7115c44c_08ae_428f_8f3c_bfe7342a7d1b.slice/crio-223826d62dc02cc99aae231ad5fedca1d4812c4381ee18826f1b85d3175cd21e WatchSource:0}: Error finding container 223826d62dc02cc99aae231ad5fedca1d4812c4381ee18826f1b85d3175cd21e: Status 404 returned error can't find the container with id 223826d62dc02cc99aae231ad5fedca1d4812c4381ee18826f1b85d3175cd21e
Sep 07 22:51:02.795295 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:02.795200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj" event={"ID":"7115c44c-08ae-428f-8f3c-bfe7342a7d1b","Type":"ContainerStarted","Data":"223826d62dc02cc99aae231ad5fedca1d4812c4381ee18826f1b85d3175cd21e"}
Sep 07 22:51:03.802686 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:03.802650    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj" event={"ID":"7115c44c-08ae-428f-8f3c-bfe7342a7d1b","Type":"ContainerStarted","Data":"62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819"}
Sep 07 22:51:03.803075 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:03.802732    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:03.817873 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:03.817821    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj" podStartSLOduration=2.476432761 podStartE2EDuration="2.81780565s" totalImagesPullingTime="341.372889ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:03.816658112 +0000 UTC m=+1130.282815109" watchObservedRunningTime="2026-09-07 22:51:03.81780565 +0000 UTC m=+1130.283962645"
Sep 07 22:51:05.753371 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:05.753336    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 22:51:06.619243 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:51:06.619205    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79245e47_d00e_40b1_b37b_c9a6300ea7f5.slice/crio-conmon-abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec.scope\": RecentStats: unable to find data in memory cache]"
Sep 07 22:51:09.813902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:09.813872    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:51:20.969714 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:20.969679    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"]
Sep 07 22:51:20.974649 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:20.974625    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:20.976805 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:20.976781    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w3-7823cc-serving-cert\""
Sep 07 22:51:20.986002 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:20.985975    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"]
Sep 07 22:51:21.054464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.054425    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3339f274-c827-4374-8613-963e27cd8340-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.054464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.054465    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.054689 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.054484    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.054689 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.054558    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6crf\" (UniqueName: \"kubernetes.io/projected/3339f274-c827-4374-8613-963e27cd8340-kube-api-access-w6crf\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.155533 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.155489    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3339f274-c827-4374-8613-963e27cd8340-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.155533 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.155534    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.155805 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.155557    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.155805 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.155594    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6crf\" (UniqueName: \"kubernetes.io/projected/3339f274-c827-4374-8613-963e27cd8340-kube-api-access-w6crf\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.156254 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.156226    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3339f274-c827-4374-8613-963e27cd8340-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.158033 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.158012    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-metrics-certs\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.158183 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.158064    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-maas-api-tls\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.163233 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.163208    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6crf\" (UniqueName: \"kubernetes.io/projected/3339f274-c827-4374-8613-963e27cd8340-kube-api-access-w6crf\") pod \"maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.286546 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.286451    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:21.566591 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.566556    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"]
Sep 07 22:51:21.568383 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:51:21.568349    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3339f274_c827_4374_8613_963e27cd8340.slice/crio-145e50d20879892cc826a75b89b0958fae81ec74f434b57e28268cba89356a05 WatchSource:0}: Error finding container 145e50d20879892cc826a75b89b0958fae81ec74f434b57e28268cba89356a05: Status 404 returned error can't find the container with id 145e50d20879892cc826a75b89b0958fae81ec74f434b57e28268cba89356a05
Sep 07 22:51:21.907445 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:21.907360    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6" event={"ID":"3339f274-c827-4374-8613-963e27cd8340","Type":"ContainerStarted","Data":"145e50d20879892cc826a75b89b0958fae81ec74f434b57e28268cba89356a05"}
Sep 07 22:51:22.914779 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:22.914741    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6" event={"ID":"3339f274-c827-4374-8613-963e27cd8340","Type":"ContainerStarted","Data":"3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f"}
Sep 07 22:51:22.915647 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:22.914936    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:22.931732 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:22.931678    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6" podStartSLOduration=2.589954271 podStartE2EDuration="2.93166181s" totalImagesPullingTime="341.707539ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:22.929327427 +0000 UTC m=+1149.395484422" watchObservedRunningTime="2026-09-07 22:51:22.93166181 +0000 UTC m=+1149.397818804"
Sep 07 22:51:28.926118 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:28.926080    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 22:51:44.818073 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.817996    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"]
Sep 07 22:51:44.821495 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.821473    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:44.823692 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.823666    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-33bf198b\""
Sep 07 22:51:44.831415 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.831390    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"]
Sep 07 22:51:44.844964 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.844928    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"]
Sep 07 22:51:44.848468 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.848445    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:44.850847 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.850827    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-33bf198b-dockercfg-h97dx\""
Sep 07 22:51:44.857030 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.857003    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"]
Sep 07 22:51:44.889024 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.888983    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86a15ad-0440-4cd1-9218-6da42187f1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:44.889024 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.889020    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qw2d\" (UniqueName: \"kubernetes.io/projected/b86a15ad-0440-4cd1-9218-6da42187f1af-kube-api-access-5qw2d\") pod \"payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:44.990367 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.990325    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mlbq7\" (UniqueName: \"kubernetes.io/projected/fac5f227-987e-46fa-b89e-aa719168d059-kube-api-access-mlbq7\") pod \"payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") " pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:44.990558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.990439    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86a15ad-0440-4cd1-9218-6da42187f1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:44.990558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.990472    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qw2d\" (UniqueName: \"kubernetes.io/projected/b86a15ad-0440-4cd1-9218-6da42187f1af-kube-api-access-5qw2d\") pod \"payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:44.990558 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.990525    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5f227-987e-46fa-b89e-aa719168d059-plugins-config-volume\") pod \"payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") " pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:44.991183 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.991163    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86a15ad-0440-4cd1-9218-6da42187f1af-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:44.997948 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:44.997919    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qw2d\" (UniqueName: \"kubernetes.io/projected/b86a15ad-0440-4cd1-9218-6da42187f1af-kube-api-access-5qw2d\") pod \"payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:45.091798 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.091672    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5f227-987e-46fa-b89e-aa719168d059-plugins-config-volume\") pod \"payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") " pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:45.091798 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.091764    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mlbq7\" (UniqueName: \"kubernetes.io/projected/fac5f227-987e-46fa-b89e-aa719168d059-kube-api-access-mlbq7\") pod \"payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") " pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:45.092434 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.092412    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5f227-987e-46fa-b89e-aa719168d059-plugins-config-volume\") pod \"payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") " pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:45.099329 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.099299    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mlbq7\" (UniqueName: \"kubernetes.io/projected/fac5f227-987e-46fa-b89e-aa719168d059-kube-api-access-mlbq7\") pod \"payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") " pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:45.134152 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.134115    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:45.162384 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.162350    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:45.406507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.406478    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"]
Sep 07 22:51:45.407593 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:51:45.407570    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb86a15ad_0440_4cd1_9218_6da42187f1af.slice/crio-302c1fd4131fdf0aff80da38913670be1dcacec72d81b9fe9025910ebea15abc WatchSource:0}: Error finding container 302c1fd4131fdf0aff80da38913670be1dcacec72d81b9fe9025910ebea15abc: Status 404 returned error can't find the container with id 302c1fd4131fdf0aff80da38913670be1dcacec72d81b9fe9025910ebea15abc
Sep 07 22:51:45.447344 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:45.447316    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"]
Sep 07 22:51:45.449108 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:51:45.449082    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac5f227_987e_46fa_b89e_aa719168d059.slice/crio-f214e8df1b14527b1b95fe07c37c998c1423bf7eed4ddced0b9144345f4844d3 WatchSource:0}: Error finding container f214e8df1b14527b1b95fe07c37c998c1423bf7eed4ddced0b9144345f4844d3: Status 404 returned error can't find the container with id f214e8df1b14527b1b95fe07c37c998c1423bf7eed4ddced0b9144345f4844d3
Sep 07 22:51:46.043657 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.043604    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb" event={"ID":"fac5f227-987e-46fa-b89e-aa719168d059","Type":"ContainerStarted","Data":"584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff"}
Sep 07 22:51:46.044102 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.043709    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb" event={"ID":"fac5f227-987e-46fa-b89e-aa719168d059","Type":"ContainerStarted","Data":"f214e8df1b14527b1b95fe07c37c998c1423bf7eed4ddced0b9144345f4844d3"}
Sep 07 22:51:46.044102 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.043747    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:46.045462 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.045437    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k" event={"ID":"b86a15ad-0440-4cd1-9218-6da42187f1af","Type":"ContainerStarted","Data":"5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70"}
Sep 07 22:51:46.045612 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.045484    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:51:46.045612 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.045503    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k" event={"ID":"b86a15ad-0440-4cd1-9218-6da42187f1af","Type":"ContainerStarted","Data":"302c1fd4131fdf0aff80da38913670be1dcacec72d81b9fe9025910ebea15abc"}
Sep 07 22:51:46.061930 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.061634    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb" podStartSLOduration=2.061614573 podStartE2EDuration="2.061614573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:46.060805304 +0000 UTC m=+1172.526962301" watchObservedRunningTime="2026-09-07 22:51:46.061614573 +0000 UTC m=+1172.527771573"
Sep 07 22:51:46.076581 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:46.076527    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k" podStartSLOduration=2.076508712 podStartE2EDuration="2.076508712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:51:46.074713347 +0000 UTC m=+1172.540870345" watchObservedRunningTime="2026-09-07 22:51:46.076508712 +0000 UTC m=+1172.542666078"
Sep 07 22:51:57.053137 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:57.053101    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:51:57.053649 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:51:57.053179    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:53:33.587998 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.587962    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"]
Sep 07 22:53:33.588551 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.588178    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb" podUID="fac5f227-987e-46fa-b89e-aa719168d059" containerName="payload-processing" containerID="cri-o://584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff" gracePeriod=30
Sep 07 22:53:33.594116 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.594084    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"]
Sep 07 22:53:33.594735 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.594405    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k" podUID="b86a15ad-0440-4cd1-9218-6da42187f1af" containerName="payload-pre-processing" containerID="cri-o://5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70" gracePeriod=30
Sep 07 22:53:33.881202 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.881173    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:53:33.884235 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.884207    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:53:33.947643 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.947598    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86a15ad-0440-4cd1-9218-6da42187f1af-plugins-config-volume\") pod \"b86a15ad-0440-4cd1-9218-6da42187f1af\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") "
Sep 07 22:53:33.947860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.947667    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mlbq7\" (UniqueName: \"kubernetes.io/projected/fac5f227-987e-46fa-b89e-aa719168d059-kube-api-access-mlbq7\") pod \"fac5f227-987e-46fa-b89e-aa719168d059\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") "
Sep 07 22:53:33.947860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.947692    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5f227-987e-46fa-b89e-aa719168d059-plugins-config-volume\") pod \"fac5f227-987e-46fa-b89e-aa719168d059\" (UID: \"fac5f227-987e-46fa-b89e-aa719168d059\") "
Sep 07 22:53:33.947860 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.947742    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qw2d\" (UniqueName: \"kubernetes.io/projected/b86a15ad-0440-4cd1-9218-6da42187f1af-kube-api-access-5qw2d\") pod \"b86a15ad-0440-4cd1-9218-6da42187f1af\" (UID: \"b86a15ad-0440-4cd1-9218-6da42187f1af\") "
Sep 07 22:53:33.948171 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.948141    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b86a15ad-0440-4cd1-9218-6da42187f1af-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b86a15ad-0440-4cd1-9218-6da42187f1af" (UID: "b86a15ad-0440-4cd1-9218-6da42187f1af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:53:33.948239 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.948163    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac5f227-987e-46fa-b89e-aa719168d059-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fac5f227-987e-46fa-b89e-aa719168d059" (UID: "fac5f227-987e-46fa-b89e-aa719168d059"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:53:33.950064 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.950034    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b86a15ad-0440-4cd1-9218-6da42187f1af-kube-api-access-5qw2d" (OuterVolumeSpecName: "kube-api-access-5qw2d") pod "b86a15ad-0440-4cd1-9218-6da42187f1af" (UID: "b86a15ad-0440-4cd1-9218-6da42187f1af"). InnerVolumeSpecName "kube-api-access-5qw2d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:53:33.950203 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:33.950175    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac5f227-987e-46fa-b89e-aa719168d059-kube-api-access-mlbq7" (OuterVolumeSpecName: "kube-api-access-mlbq7") pod "fac5f227-987e-46fa-b89e-aa719168d059" (UID: "fac5f227-987e-46fa-b89e-aa719168d059"). InnerVolumeSpecName "kube-api-access-mlbq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:53:34.049269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.049223    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qw2d\" (UniqueName: \"kubernetes.io/projected/b86a15ad-0440-4cd1-9218-6da42187f1af-kube-api-access-5qw2d\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:53:34.049269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.049258    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b86a15ad-0440-4cd1-9218-6da42187f1af-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:53:34.049269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.049268    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mlbq7\" (UniqueName: \"kubernetes.io/projected/fac5f227-987e-46fa-b89e-aa719168d059-kube-api-access-mlbq7\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:53:34.049269 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.049278    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fac5f227-987e-46fa-b89e-aa719168d059-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:53:34.651595 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.651554    2582 generic.go:356] "Generic (PLEG): container finished" podID="fac5f227-987e-46fa-b89e-aa719168d059" containerID="584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff" exitCode=0
Sep 07 22:53:34.652085 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.651604    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"
Sep 07 22:53:34.652085 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.651643    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb" event={"ID":"fac5f227-987e-46fa-b89e-aa719168d059","Type":"ContainerDied","Data":"584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff"}
Sep 07 22:53:34.652085 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.651714    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb" event={"ID":"fac5f227-987e-46fa-b89e-aa719168d059","Type":"ContainerDied","Data":"f214e8df1b14527b1b95fe07c37c998c1423bf7eed4ddced0b9144345f4844d3"}
Sep 07 22:53:34.652085 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.651751    2582 scope.go:117] "RemoveContainer" containerID="584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff"
Sep 07 22:53:34.653229 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.653044    2582 generic.go:356] "Generic (PLEG): container finished" podID="b86a15ad-0440-4cd1-9218-6da42187f1af" containerID="5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70" exitCode=0
Sep 07 22:53:34.653229 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.653090    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"
Sep 07 22:53:34.653229 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.653127    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k" event={"ID":"b86a15ad-0440-4cd1-9218-6da42187f1af","Type":"ContainerDied","Data":"5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70"}
Sep 07 22:53:34.653229 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.653171    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k" event={"ID":"b86a15ad-0440-4cd1-9218-6da42187f1af","Type":"ContainerDied","Data":"302c1fd4131fdf0aff80da38913670be1dcacec72d81b9fe9025910ebea15abc"}
Sep 07 22:53:34.661196 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.661176    2582 scope.go:117] "RemoveContainer" containerID="584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff"
Sep 07 22:53:34.661608 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:53:34.661588    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff\": container with ID starting with 584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff not found: ID does not exist" containerID="584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff"
Sep 07 22:53:34.661702 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.661616    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff"} err="failed to get container status \"584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff\": rpc error: code = NotFound desc = could not find container \"584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff\": container with ID starting with 584b12802bb76b37bf7a081b59aa0a732ca861c13946ab21fcc61746090d76ff not found: ID does not exist"
Sep 07 22:53:34.661702 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.661634    2582 scope.go:117] "RemoveContainer" containerID="5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70"
Sep 07 22:53:34.670023 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.670009    2582 scope.go:117] "RemoveContainer" containerID="5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70"
Sep 07 22:53:34.670341 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:53:34.670324    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70\": container with ID starting with 5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70 not found: ID does not exist" containerID="5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70"
Sep 07 22:53:34.670433 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.670350    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70"} err="failed to get container status \"5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70\": rpc error: code = NotFound desc = could not find container \"5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70\": container with ID starting with 5fde4cd4927c4d790a51bce69649e39898d509b896d369bedcc61edfb7c67e70 not found: ID does not exist"
Sep 07 22:53:34.671502 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.671484    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"]
Sep 07 22:53:34.675372 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.675342    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-33bf198b-56bdd4d4f8-84h6k"]
Sep 07 22:53:34.684510 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.684488    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"]
Sep 07 22:53:34.687973 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:34.687950    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-33bf198b-894cb5dbf-m9qcb"]
Sep 07 22:53:36.089606 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:36.089569    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b86a15ad-0440-4cd1-9218-6da42187f1af" path="/var/lib/kubelet/pods/b86a15ad-0440-4cd1-9218-6da42187f1af/volumes"
Sep 07 22:53:36.090004 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:53:36.089989    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fac5f227-987e-46fa-b89e-aa719168d059" path="/var/lib/kubelet/pods/fac5f227-987e-46fa-b89e-aa719168d059/volumes"
Sep 07 22:54:14.162530 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:54:14.162495    2582 scope.go:117] "RemoveContainer" containerID="1f8a83dfc5dcbea306fe0b28527e3effbbcacc0a379c471986d96619539a21c5"
Sep 07 22:55:23.466607 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.466560    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"]
Sep 07 22:55:23.467202 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.467109    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac5f227-987e-46fa-b89e-aa719168d059" containerName="payload-processing"
Sep 07 22:55:23.467202 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.467132    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac5f227-987e-46fa-b89e-aa719168d059" containerName="payload-processing"
Sep 07 22:55:23.467202 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.467186    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b86a15ad-0440-4cd1-9218-6da42187f1af" containerName="payload-pre-processing"
Sep 07 22:55:23.467202 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.467195    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b86a15ad-0440-4cd1-9218-6da42187f1af" containerName="payload-pre-processing"
Sep 07 22:55:23.467456 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.467318    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b86a15ad-0440-4cd1-9218-6da42187f1af" containerName="payload-pre-processing"
Sep 07 22:55:23.467456 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.467339    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac5f227-987e-46fa-b89e-aa719168d059" containerName="payload-processing"
Sep 07 22:55:23.470506 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.470483    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.472864 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.472845    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"kube-root-ca.crt\""
Sep 07 22:55:23.472988 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.472847    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"default-dockercfg-cmstf\""
Sep 07 22:55:23.473750 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.473726    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"sub-test-model-1d0395-kserve-self-signed-certs\""
Sep 07 22:55:23.473750 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.473731    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-1d0395\"/\"openshift-service-ca.crt\""
Sep 07 22:55:23.480143 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.480117    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"]
Sep 07 22:55:23.550193 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.550156    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-model-cache\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.550404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.550211    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-home\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.550404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.550309    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-dshm\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.550404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.550337    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-tmp-dir\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.550404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.550370    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1799985-3dcd-45c7-9f8f-68a228d8614a-tls-certs\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.550404 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.550387    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qj7g\" (UniqueName: \"kubernetes.io/projected/c1799985-3dcd-45c7-9f8f-68a228d8614a-kube-api-access-4qj7g\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.651974 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.651934    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-model-cache\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652002    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-home\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652044    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-dshm\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652064    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-tmp-dir\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652094    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1799985-3dcd-45c7-9f8f-68a228d8614a-tls-certs\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652167    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qj7g\" (UniqueName: \"kubernetes.io/projected/c1799985-3dcd-45c7-9f8f-68a228d8614a-kube-api-access-4qj7g\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652611 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652452    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-model-cache\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652611 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652507    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-home\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.652611 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.652550    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-tmp-dir\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.654619 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.654588    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-dshm\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.654756 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.654738    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1799985-3dcd-45c7-9f8f-68a228d8614a-tls-certs\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.659713 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.659689    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qj7g\" (UniqueName: \"kubernetes.io/projected/c1799985-3dcd-45c7-9f8f-68a228d8614a-kube-api-access-4qj7g\") pod \"sub-test-model-1d0395-kserve-688d5b8dcf-26hrw\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") " pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:23.781356 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:23.781253    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:24.051670 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:24.051640    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"]
Sep 07 22:55:24.053784 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:55:24.053755    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc1799985_3dcd_45c7_9f8f_68a228d8614a.slice/crio-289208462dad37855456409b0685b83998112b36dd8bdc9bc5b5e711b0a23239 WatchSource:0}: Error finding container 289208462dad37855456409b0685b83998112b36dd8bdc9bc5b5e711b0a23239: Status 404 returned error can't find the container with id 289208462dad37855456409b0685b83998112b36dd8bdc9bc5b5e711b0a23239
Sep 07 22:55:24.234912 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:24.234873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw" event={"ID":"c1799985-3dcd-45c7-9f8f-68a228d8614a","Type":"ContainerStarted","Data":"4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d"}
Sep 07 22:55:24.235085 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:24.234921    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw" event={"ID":"c1799985-3dcd-45c7-9f8f-68a228d8614a","Type":"ContainerStarted","Data":"289208462dad37855456409b0685b83998112b36dd8bdc9bc5b5e711b0a23239"}
Sep 07 22:55:24.251038 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:24.250989    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw" podStartSLOduration=1.250973682 podStartE2EDuration="1.250973682s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:55:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:55:24.249417908 +0000 UTC m=+1390.715574905" watchObservedRunningTime="2026-09-07 22:55:24.250973682 +0000 UTC m=+1390.717130676"
Sep 07 22:55:33.782188 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:33.782147    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:33.782708 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:33.782199    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:33.794547 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:33.794522    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:34.298030 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:34.297995    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:55:54.325311 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.325264    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"]
Sep 07 22:55:54.328721 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.328702    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.332222 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.332193    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-dd5b43\""
Sep 07 22:55:54.332354 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.332244    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-dd5b43-dockercfg-wpnwr\""
Sep 07 22:55:54.349932 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.349905    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"]
Sep 07 22:55:54.450830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.450795    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.450830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.450829    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8npf2\" (UniqueName: \"kubernetes.io/projected/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-kube-api-access-8npf2\") pod \"payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.552010 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.551974    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.552010 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.552010    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8npf2\" (UniqueName: \"kubernetes.io/projected/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-kube-api-access-8npf2\") pod \"payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.552726 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.552706    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-plugins-config-volume\") pod \"payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.560676 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.560649    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8npf2\" (UniqueName: \"kubernetes.io/projected/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-kube-api-access-8npf2\") pod \"payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") " pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.639597 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.639508    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:54.910872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:54.910844    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"]
Sep 07 22:55:54.912730 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:55:54.912699    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f6fe0c5_a47d_4a26_a0e9_c2409e46034f.slice/crio-9683a2f24b8b3ecb936c7484aa86dde1de62585669b1b739e6a85e86a587022f WatchSource:0}: Error finding container 9683a2f24b8b3ecb936c7484aa86dde1de62585669b1b739e6a85e86a587022f: Status 404 returned error can't find the container with id 9683a2f24b8b3ecb936c7484aa86dde1de62585669b1b739e6a85e86a587022f
Sep 07 22:55:55.398954 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:55.398916    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg" event={"ID":"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f","Type":"ContainerStarted","Data":"dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7"}
Sep 07 22:55:55.399461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:55.398971    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg" event={"ID":"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f","Type":"ContainerStarted","Data":"9683a2f24b8b3ecb936c7484aa86dde1de62585669b1b739e6a85e86a587022f"}
Sep 07 22:55:55.399461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:55.399003    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:55:55.415571 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:55:55.415510    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg" podStartSLOduration=1.415490344 podStartE2EDuration="1.415490344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:55:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:55:55.412608599 +0000 UTC m=+1421.878765597" watchObservedRunningTime="2026-09-07 22:55:55.415490344 +0000 UTC m=+1421.881647340"
Sep 07 22:56:06.407148 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:06.407060    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:56:14.198270 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:14.198235    2582 scope.go:117] "RemoveContainer" containerID="abcfeb006d760b83378d43a4852a5a2c879a60a3d7474f2500aba8028efc2aec"
Sep 07 22:56:22.879946 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.879909    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"]
Sep 07 22:56:22.884837 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.884815    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:22.887301 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.887257    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c7776f-serving-cert\""
Sep 07 22:56:22.898210 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.898171    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"]
Sep 07 22:56:22.941599 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.941552    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dbab49-36a5-4276-8609-4efb9d722c22-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:22.941599 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.941595    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jjt6\" (UniqueName: \"kubernetes.io/projected/e5dbab49-36a5-4276-8609-4efb9d722c22-kube-api-access-7jjt6\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:22.941838 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.941624    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:22.941838 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:22.941723    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-metrics-certs\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.042763 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.042707    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.042951 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.042781    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-metrics-certs\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.042951 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.042862    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dbab49-36a5-4276-8609-4efb9d722c22-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.042951 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.042880    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jjt6\" (UniqueName: \"kubernetes.io/projected/e5dbab49-36a5-4276-8609-4efb9d722c22-kube-api-access-7jjt6\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.042951 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:56:23.042881    2582 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-serving-cert: secret "maas-api-e2e-api-b-c7776f-serving-cert" not found
Sep 07 22:56:23.043099 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:56:23.042988    2582 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls podName:e5dbab49-36a5-4276-8609-4efb9d722c22 nodeName:}" failed. No retries permitted until 2026-09-07 22:56:23.542962795 +0000 UTC m=+1450.009119781 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls") pod "maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq" (UID: "e5dbab49-36a5-4276-8609-4efb9d722c22") : secret "maas-api-e2e-api-b-c7776f-serving-cert" not found
Sep 07 22:56:23.043666 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.043642    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dbab49-36a5-4276-8609-4efb9d722c22-openshift-service-ca\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.045367 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.045345    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-metrics-certs\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.058072 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.058045    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jjt6\" (UniqueName: \"kubernetes.io/projected/e5dbab49-36a5-4276-8609-4efb9d722c22-kube-api-access-7jjt6\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.547525 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.547488    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.550156 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.550127    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls\") pod \"maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:23.797732 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:23.797642    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:24.072198 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:24.072163    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"]
Sep 07 22:56:24.076568 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:56:24.076530    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dbab49_36a5_4276_8609_4efb9d722c22.slice/crio-73c2ed5cfbf8331094be8247ba5239f5fa034ae6c1514937814786df58ac9bee WatchSource:0}: Error finding container 73c2ed5cfbf8331094be8247ba5239f5fa034ae6c1514937814786df58ac9bee: Status 404 returned error can't find the container with id 73c2ed5cfbf8331094be8247ba5239f5fa034ae6c1514937814786df58ac9bee
Sep 07 22:56:24.078249 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:24.078227    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 22:56:24.555945 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:24.555907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq" event={"ID":"e5dbab49-36a5-4276-8609-4efb9d722c22","Type":"ContainerStarted","Data":"e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569"}
Sep 07 22:56:24.556147 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:24.555965    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq" event={"ID":"e5dbab49-36a5-4276-8609-4efb9d722c22","Type":"ContainerStarted","Data":"73c2ed5cfbf8331094be8247ba5239f5fa034ae6c1514937814786df58ac9bee"}
Sep 07 22:56:24.556147 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:24.555993    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:24.571314 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:24.571241    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq" podStartSLOduration=2.193630665 podStartE2EDuration="2.571225481s" totalImagesPullingTime="377.594816ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:56:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:56:24.570776512 +0000 UTC m=+1451.036933504" watchObservedRunningTime="2026-09-07 22:56:24.571225481 +0000 UTC m=+1451.037382476"
Sep 07 22:56:30.568988 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:30.568954    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:56:35.059255 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:35.059217    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"]
Sep 07 22:56:50.079314 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.079228    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw" podUID="c1799985-3dcd-45c7-9f8f-68a228d8614a" containerName="main" containerID="cri-o://4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d" gracePeriod=45
Sep 07 22:56:50.334744 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.334680    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:56:50.522830 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.522795    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-tmp-dir\") pod \"c1799985-3dcd-45c7-9f8f-68a228d8614a\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") "
Sep 07 22:56:50.523001 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.522862    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-model-cache\") pod \"c1799985-3dcd-45c7-9f8f-68a228d8614a\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") "
Sep 07 22:56:50.523001 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.522890    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-home\") pod \"c1799985-3dcd-45c7-9f8f-68a228d8614a\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") "
Sep 07 22:56:50.523001 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.522960    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-dshm\") pod \"c1799985-3dcd-45c7-9f8f-68a228d8614a\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") "
Sep 07 22:56:50.523165 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523002    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1799985-3dcd-45c7-9f8f-68a228d8614a-tls-certs\") pod \"c1799985-3dcd-45c7-9f8f-68a228d8614a\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") "
Sep 07 22:56:50.523165 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523029    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4qj7g\" (UniqueName: \"kubernetes.io/projected/c1799985-3dcd-45c7-9f8f-68a228d8614a-kube-api-access-4qj7g\") pod \"c1799985-3dcd-45c7-9f8f-68a228d8614a\" (UID: \"c1799985-3dcd-45c7-9f8f-68a228d8614a\") "
Sep 07 22:56:50.523165 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523134    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c1799985-3dcd-45c7-9f8f-68a228d8614a" (UID: "c1799985-3dcd-45c7-9f8f-68a228d8614a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:50.523357 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523175    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-model-cache" (OuterVolumeSpecName: "model-cache") pod "c1799985-3dcd-45c7-9f8f-68a228d8614a" (UID: "c1799985-3dcd-45c7-9f8f-68a228d8614a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:50.523420 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523347    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-home" (OuterVolumeSpecName: "home") pod "c1799985-3dcd-45c7-9f8f-68a228d8614a" (UID: "c1799985-3dcd-45c7-9f8f-68a228d8614a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:50.523491 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523479    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-tmp-dir\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:56:50.523551 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523499    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-model-cache\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:56:50.523551 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.523513    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-home\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:56:50.525450 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.525403    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-dshm" (OuterVolumeSpecName: "dshm") pod "c1799985-3dcd-45c7-9f8f-68a228d8614a" (UID: "c1799985-3dcd-45c7-9f8f-68a228d8614a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:56:50.525450 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.525404    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1799985-3dcd-45c7-9f8f-68a228d8614a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c1799985-3dcd-45c7-9f8f-68a228d8614a" (UID: "c1799985-3dcd-45c7-9f8f-68a228d8614a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:56:50.525608 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.525508    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1799985-3dcd-45c7-9f8f-68a228d8614a-kube-api-access-4qj7g" (OuterVolumeSpecName: "kube-api-access-4qj7g") pod "c1799985-3dcd-45c7-9f8f-68a228d8614a" (UID: "c1799985-3dcd-45c7-9f8f-68a228d8614a"). InnerVolumeSpecName "kube-api-access-4qj7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:56:50.625032 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.624937    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c1799985-3dcd-45c7-9f8f-68a228d8614a-dshm\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:56:50.625032 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.624970    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c1799985-3dcd-45c7-9f8f-68a228d8614a-tls-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:56:50.625032 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.624982    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qj7g\" (UniqueName: \"kubernetes.io/projected/c1799985-3dcd-45c7-9f8f-68a228d8614a-kube-api-access-4qj7g\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:56:50.699365 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.699327    2582 generic.go:356] "Generic (PLEG): container finished" podID="c1799985-3dcd-45c7-9f8f-68a228d8614a" containerID="4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d" exitCode=0
Sep 07 22:56:50.699532 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.699372    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"
Sep 07 22:56:50.699532 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.699380    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw" event={"ID":"c1799985-3dcd-45c7-9f8f-68a228d8614a","Type":"ContainerDied","Data":"4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d"}
Sep 07 22:56:50.699532 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.699414    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw" event={"ID":"c1799985-3dcd-45c7-9f8f-68a228d8614a","Type":"ContainerDied","Data":"289208462dad37855456409b0685b83998112b36dd8bdc9bc5b5e711b0a23239"}
Sep 07 22:56:50.699532 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.699434    2582 scope.go:117] "RemoveContainer" containerID="4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d"
Sep 07 22:56:50.709493 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.709475    2582 scope.go:117] "RemoveContainer" containerID="4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d"
Sep 07 22:56:50.709801 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:56:50.709778    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d\": container with ID starting with 4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d not found: ID does not exist" containerID="4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d"
Sep 07 22:56:50.709852 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.709810    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d"} err="failed to get container status \"4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d\": rpc error: code = NotFound desc = could not find container \"4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d\": container with ID starting with 4f375fbb9ca1222586040e91e887e3d7dcf242d3b7db9ffb0d751129f027987d not found: ID does not exist"
Sep 07 22:56:50.721910 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.721880    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"]
Sep 07 22:56:50.735424 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:50.735375    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-1d0395/sub-test-model-1d0395-kserve-688d5b8dcf-26hrw"]
Sep 07 22:56:52.090982 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:56:52.090942    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1799985-3dcd-45c7-9f8f-68a228d8614a" path="/var/lib/kubelet/pods/c1799985-3dcd-45c7-9f8f-68a228d8614a/volumes"
Sep 07 22:57:04.965925 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:04.965892    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"]
Sep 07 22:57:04.966562 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:04.966137    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg" podUID="1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" containerName="payload-processing" containerID="cri-o://dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7" gracePeriod=30
Sep 07 22:57:05.251050 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.251010    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:57:05.374009 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.373969    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-plugins-config-volume\") pod \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") "
Sep 07 22:57:05.374224 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.374096    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8npf2\" (UniqueName: \"kubernetes.io/projected/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-kube-api-access-8npf2\") pod \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\" (UID: \"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f\") "
Sep 07 22:57:05.374482 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.374455    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" (UID: "1f6fe0c5-a47d-4a26-a0e9-c2409e46034f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:05.376349 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.376320    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-kube-api-access-8npf2" (OuterVolumeSpecName: "kube-api-access-8npf2") pod "1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" (UID: "1f6fe0c5-a47d-4a26-a0e9-c2409e46034f"). InnerVolumeSpecName "kube-api-access-8npf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:05.474985 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.474942    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8npf2\" (UniqueName: \"kubernetes.io/projected/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-kube-api-access-8npf2\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.474985 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.474980    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:05.781813 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.781777    2582 generic.go:356] "Generic (PLEG): container finished" podID="1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" containerID="dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7" exitCode=0
Sep 07 22:57:05.781989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.781823    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"
Sep 07 22:57:05.781989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.781864    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg" event={"ID":"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f","Type":"ContainerDied","Data":"dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7"}
Sep 07 22:57:05.781989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.781948    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg" event={"ID":"1f6fe0c5-a47d-4a26-a0e9-c2409e46034f","Type":"ContainerDied","Data":"9683a2f24b8b3ecb936c7484aa86dde1de62585669b1b739e6a85e86a587022f"}
Sep 07 22:57:05.781989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.781979    2582 scope.go:117] "RemoveContainer" containerID="dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7"
Sep 07 22:57:05.791990 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.791968    2582 scope.go:117] "RemoveContainer" containerID="dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7"
Sep 07 22:57:05.792379 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:57:05.792358    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7\": container with ID starting with dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7 not found: ID does not exist" containerID="dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7"
Sep 07 22:57:05.792438 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.792391    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7"} err="failed to get container status \"dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7\": rpc error: code = NotFound desc = could not find container \"dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7\": container with ID starting with dfd733c24e07d13f23ed44c065c913e208490c1fa01b97f76e3764c31e22b0e7 not found: ID does not exist"
Sep 07 22:57:05.804114 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.804082    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"]
Sep 07 22:57:05.807871 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:05.807844    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-dd5b43-7bddd656dc-4dcmg"]
Sep 07 22:57:06.091077 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:06.091041    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" path="/var/lib/kubelet/pods/1f6fe0c5-a47d-4a26-a0e9-c2409e46034f/volumes"
Sep 07 22:57:17.869647 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.869604    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"]
Sep 07 22:57:17.870008 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.869979    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1799985-3dcd-45c7-9f8f-68a228d8614a" containerName="main"
Sep 07 22:57:17.870008 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.869991    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1799985-3dcd-45c7-9f8f-68a228d8614a" containerName="main"
Sep 07 22:57:17.870008 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.870002    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" containerName="payload-processing"
Sep 07 22:57:17.870008 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.870009    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" containerName="payload-processing"
Sep 07 22:57:17.870136 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.870071    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1799985-3dcd-45c7-9f8f-68a228d8614a" containerName="main"
Sep 07 22:57:17.870136 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.870081    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f6fe0c5-a47d-4a26-a0e9-c2409e46034f" containerName="payload-processing"
Sep 07 22:57:17.874521 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.874501    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:17.877805 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.877784    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"openshift-service-ca.crt\""
Sep 07 22:57:17.877805 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.877796    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"kube-root-ca.crt\""
Sep 07 22:57:17.877980 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.877784    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"default-dockercfg-6pjx2\""
Sep 07 22:57:17.877980 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.877785    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-7823cc\"/\"rate-test-model-7823cc-kserve-self-signed-certs\""
Sep 07 22:57:17.883806 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:57:17.883782    2582 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a7be0ba_7b27_4e29_8b56_98f1a8b70f8b.slice\": RecentStats: unable to find data in memory cache]"
Sep 07 22:57:17.884364 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.884339    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"]
Sep 07 22:57:17.998863 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.998822    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tls-certs\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:17.998863 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.998874    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tmp-dir\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:17.999088 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.998920    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-dshm\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:17.999088 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.998961    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-home\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:17.999088 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.999030    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-model-cache\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:17.999194 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:17.999086    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cm8cp\" (UniqueName: \"kubernetes.io/projected/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-kube-api-access-cm8cp\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.099776 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.099740    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tls-certs\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.099936 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.099786    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tmp-dir\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.099936 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.099824    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-dshm\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.099936 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.099843    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-home\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.099936 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.099874    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-model-cache\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.099936 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.099933    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cm8cp\" (UniqueName: \"kubernetes.io/projected/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-kube-api-access-cm8cp\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.100385 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.100352    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tmp-dir\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.100385 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.100370    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-model-cache\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.100538 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.100405    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-home\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.102071 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.102046    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-dshm\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.102348 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.102330    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tls-certs\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.107146 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.107126    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cm8cp\" (UniqueName: \"kubernetes.io/projected/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-kube-api-access-cm8cp\") pod \"rate-test-model-7823cc-kserve-6965bbdc47-wqtn8\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") " pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.186337 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.186209    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:18.460899 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.460869    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"]
Sep 07 22:57:18.462820 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:57:18.462789    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2a7be0ba_7b27_4e29_8b56_98f1a8b70f8b.slice/crio-0d3ee835e962ba0742a9cd9b12eb39b08534a8aa3da8ef450467731f5b3138b5 WatchSource:0}: Error finding container 0d3ee835e962ba0742a9cd9b12eb39b08534a8aa3da8ef450467731f5b3138b5: Status 404 returned error can't find the container with id 0d3ee835e962ba0742a9cd9b12eb39b08534a8aa3da8ef450467731f5b3138b5
Sep 07 22:57:18.860073 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.860030    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8" event={"ID":"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b","Type":"ContainerStarted","Data":"9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43"}
Sep 07 22:57:18.860320 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.860089    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8" event={"ID":"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b","Type":"ContainerStarted","Data":"0d3ee835e962ba0742a9cd9b12eb39b08534a8aa3da8ef450467731f5b3138b5"}
Sep 07 22:57:18.879079 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:18.879006    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8" podStartSLOduration=1.8789811950000002 podStartE2EDuration="1.878981195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:57:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:57:18.875117848 +0000 UTC m=+1505.341274847" watchObservedRunningTime="2026-09-07 22:57:18.878981195 +0000 UTC m=+1505.345138191"
Sep 07 22:57:27.224026 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.223988    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"]
Sep 07 22:57:27.224565 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.224253    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq" podUID="e5dbab49-36a5-4276-8609-4efb9d722c22" containerName="maas-api" containerID="cri-o://e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569" gracePeriod=30
Sep 07 22:57:27.482903 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.482836    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:57:27.593084 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.593051    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls\") pod \"e5dbab49-36a5-4276-8609-4efb9d722c22\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") "
Sep 07 22:57:27.593276 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.593106    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dbab49-36a5-4276-8609-4efb9d722c22-openshift-service-ca\") pod \"e5dbab49-36a5-4276-8609-4efb9d722c22\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") "
Sep 07 22:57:27.593276 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.593143    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-metrics-certs\") pod \"e5dbab49-36a5-4276-8609-4efb9d722c22\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") "
Sep 07 22:57:27.593276 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.593186    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jjt6\" (UniqueName: \"kubernetes.io/projected/e5dbab49-36a5-4276-8609-4efb9d722c22-kube-api-access-7jjt6\") pod \"e5dbab49-36a5-4276-8609-4efb9d722c22\" (UID: \"e5dbab49-36a5-4276-8609-4efb9d722c22\") "
Sep 07 22:57:27.593637 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.593598    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dbab49-36a5-4276-8609-4efb9d722c22-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "e5dbab49-36a5-4276-8609-4efb9d722c22" (UID: "e5dbab49-36a5-4276-8609-4efb9d722c22"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:27.595518 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.595467    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5dbab49-36a5-4276-8609-4efb9d722c22-kube-api-access-7jjt6" (OuterVolumeSpecName: "kube-api-access-7jjt6") pod "e5dbab49-36a5-4276-8609-4efb9d722c22" (UID: "e5dbab49-36a5-4276-8609-4efb9d722c22"). InnerVolumeSpecName "kube-api-access-7jjt6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:27.595653 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.595513    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "e5dbab49-36a5-4276-8609-4efb9d722c22" (UID: "e5dbab49-36a5-4276-8609-4efb9d722c22"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:57:27.595653 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.595602    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "e5dbab49-36a5-4276-8609-4efb9d722c22" (UID: "e5dbab49-36a5-4276-8609-4efb9d722c22"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:57:27.694174 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.694139    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:27.694174 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.694171    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dbab49-36a5-4276-8609-4efb9d722c22-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:27.694174 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.694182    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e5dbab49-36a5-4276-8609-4efb9d722c22-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:27.694456 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.694192    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jjt6\" (UniqueName: \"kubernetes.io/projected/e5dbab49-36a5-4276-8609-4efb9d722c22-kube-api-access-7jjt6\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:27.910347 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.910306    2582 generic.go:356] "Generic (PLEG): container finished" podID="e5dbab49-36a5-4276-8609-4efb9d722c22" containerID="e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569" exitCode=0
Sep 07 22:57:27.910347 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.910341    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"
Sep 07 22:57:27.910557 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.910358    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq" event={"ID":"e5dbab49-36a5-4276-8609-4efb9d722c22","Type":"ContainerDied","Data":"e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569"}
Sep 07 22:57:27.910557 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.910432    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq" event={"ID":"e5dbab49-36a5-4276-8609-4efb9d722c22","Type":"ContainerDied","Data":"73c2ed5cfbf8331094be8247ba5239f5fa034ae6c1514937814786df58ac9bee"}
Sep 07 22:57:27.910557 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.910470    2582 scope.go:117] "RemoveContainer" containerID="e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569"
Sep 07 22:57:27.922209 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.922188    2582 scope.go:117] "RemoveContainer" containerID="e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569"
Sep 07 22:57:27.922643 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:57:27.922617    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569\": container with ID starting with e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569 not found: ID does not exist" containerID="e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569"
Sep 07 22:57:27.922736 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.922652    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569"} err="failed to get container status \"e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569\": rpc error: code = NotFound desc = could not find container \"e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569\": container with ID starting with e78bdbe5c8cee283250426de950ce79dfda4e0b5dc212e0d0f7abda058fb0569 not found: ID does not exist"
Sep 07 22:57:27.935399 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.935369    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"]
Sep 07 22:57:27.937507 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:27.937486    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c7776f-774fbf64c9-ftwgq"]
Sep 07 22:57:28.090116 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:28.090080    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5dbab49-36a5-4276-8609-4efb9d722c22" path="/var/lib/kubelet/pods/e5dbab49-36a5-4276-8609-4efb9d722c22/volumes"
Sep 07 22:57:28.186734 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:28.186645    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:28.186734 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:28.186687    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:28.199623 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:28.199590    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:28.929730 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:28.929697    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:57:44.579922 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:44.579762    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"]
Sep 07 22:57:44.580908 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:44.580048    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht" podUID="7d50b72b-1203-43b7-b38d-9170e3e6c824" containerName="payload-pre-processing" containerID="cri-o://38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad" gracePeriod=30
Sep 07 22:57:44.850463 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:44.850427    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:57:44.957794 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:44.957746    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-trsn2\" (UniqueName: \"kubernetes.io/projected/7d50b72b-1203-43b7-b38d-9170e3e6c824-kube-api-access-trsn2\") pod \"7d50b72b-1203-43b7-b38d-9170e3e6c824\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") "
Sep 07 22:57:44.957972 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:44.957807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d50b72b-1203-43b7-b38d-9170e3e6c824-plugins-config-volume\") pod \"7d50b72b-1203-43b7-b38d-9170e3e6c824\" (UID: \"7d50b72b-1203-43b7-b38d-9170e3e6c824\") "
Sep 07 22:57:44.958229 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:44.958207    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d50b72b-1203-43b7-b38d-9170e3e6c824-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7d50b72b-1203-43b7-b38d-9170e3e6c824" (UID: "7d50b72b-1203-43b7-b38d-9170e3e6c824"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:44.959933 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:44.959909    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d50b72b-1203-43b7-b38d-9170e3e6c824-kube-api-access-trsn2" (OuterVolumeSpecName: "kube-api-access-trsn2") pod "7d50b72b-1203-43b7-b38d-9170e3e6c824" (UID: "7d50b72b-1203-43b7-b38d-9170e3e6c824"). InnerVolumeSpecName "kube-api-access-trsn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:45.004856 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.004815    2582 generic.go:356] "Generic (PLEG): container finished" podID="7d50b72b-1203-43b7-b38d-9170e3e6c824" containerID="38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad" exitCode=0
Sep 07 22:57:45.005027 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.004872    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"
Sep 07 22:57:45.005027 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.004907    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht" event={"ID":"7d50b72b-1203-43b7-b38d-9170e3e6c824","Type":"ContainerDied","Data":"38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad"}
Sep 07 22:57:45.005027 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.004961    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht" event={"ID":"7d50b72b-1203-43b7-b38d-9170e3e6c824","Type":"ContainerDied","Data":"d1821f880026b81b12774b6b357eededda4ad8ab84917e1cc4de77b571d29afe"}
Sep 07 22:57:45.005027 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.004985    2582 scope.go:117] "RemoveContainer" containerID="38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad"
Sep 07 22:57:45.014659 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.014639    2582 scope.go:117] "RemoveContainer" containerID="38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad"
Sep 07 22:57:45.014998 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:57:45.014965    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad\": container with ID starting with 38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad not found: ID does not exist" containerID="38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad"
Sep 07 22:57:45.015106 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.015010    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad"} err="failed to get container status \"38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad\": rpc error: code = NotFound desc = could not find container \"38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad\": container with ID starting with 38fb819523dbaa75e55bcafea7b17f8b2249fde7c921ae750f5f5053263466ad not found: ID does not exist"
Sep 07 22:57:45.026955 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.026918    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"]
Sep 07 22:57:45.029044 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.029016    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-82409b-5d9dd4f4f8-t9mht"]
Sep 07 22:57:45.058872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.058822    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trsn2\" (UniqueName: \"kubernetes.io/projected/7d50b72b-1203-43b7-b38d-9170e3e6c824-kube-api-access-trsn2\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:45.058872 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:45.058869    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7d50b72b-1203-43b7-b38d-9170e3e6c824-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:46.090147 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:46.090108    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7d50b72b-1203-43b7-b38d-9170e3e6c824" path="/var/lib/kubelet/pods/7d50b72b-1203-43b7-b38d-9170e3e6c824/volumes"
Sep 07 22:57:50.890716 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:50.890675    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"]
Sep 07 22:57:50.891150 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:50.890911    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" containerID="cri-o://9b961c334dd604573421140e5c7a3a4873f5123b24d5a2fe7cdc01788bac1266" gracePeriod=30
Sep 07 22:57:53.913464 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:53.913425    2582 patch_prober.go:28] interesting pod/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 07 22:57:53.913950 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:53.913503    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 07 22:57:56.070914 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.070868    2582 generic.go:356] "Generic (PLEG): container finished" podID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerID="9b961c334dd604573421140e5c7a3a4873f5123b24d5a2fe7cdc01788bac1266" exitCode=0
Sep 07 22:57:56.071301 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.070941    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" event={"ID":"f0c26ae9-651d-471b-b939-6805f3c5f832","Type":"ContainerDied","Data":"9b961c334dd604573421140e5c7a3a4873f5123b24d5a2fe7cdc01788bac1266"}
Sep 07 22:57:56.071301 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.071014    2582 scope.go:117] "RemoveContainer" containerID="c61aade99215aa511d4013705416bd6175d7710207989982f32df33e87e0233f"
Sep 07 22:57:56.156458 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.156393    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:57:56.271713 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271676    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-certs\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.271909 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271810    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-envoy\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.271909 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271849    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c26ae9-651d-471b-b939-6805f3c5f832-istiod-ca-cert\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.271909 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271887    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-podinfo\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.272090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271926    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-socket\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.272090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271968    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-token\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.272090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271967    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:56.272090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.271999    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-data\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.272090 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272080    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7w257\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-kube-api-access-7w257\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.272403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272165    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-credential-socket\") pod \"f0c26ae9-651d-471b-b939-6805f3c5f832\" (UID: \"f0c26ae9-651d-471b-b939-6805f3c5f832\") "
Sep 07 22:57:56.272403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272214    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:56.272403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272241    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-data" (OuterVolumeSpecName: "istio-data") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:56.272403 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272330    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0c26ae9-651d-471b-b939-6805f3c5f832-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:57:56.272601 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:56.272713 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272692    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.272786 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272719    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-data\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.272786 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272733    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-credential-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.272786 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272748    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-workload-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.272786 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.272761    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0c26ae9-651d-471b-b939-6805f3c5f832-istiod-ca-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.274235 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.274206    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:57:56.274235 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.274221    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-kube-api-access-7w257" (OuterVolumeSpecName: "kube-api-access-7w257") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "kube-api-access-7w257". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:56.274400 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.274306    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 22:57:56.274400 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.274328    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-token" (OuterVolumeSpecName: "istio-token") pod "f0c26ae9-651d-471b-b939-6805f3c5f832" (UID: "f0c26ae9-651d-471b-b939-6805f3c5f832"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:57:56.373422 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.373384    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-envoy\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.373422 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.373414    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-podinfo\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.373422 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.373423    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-istio-token\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:56.373661 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:56.373431    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7w257\" (UniqueName: \"kubernetes.io/projected/f0c26ae9-651d-471b-b939-6805f3c5f832-kube-api-access-7w257\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:57:57.077857 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:57.077823    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k" event={"ID":"f0c26ae9-651d-471b-b939-6805f3c5f832","Type":"ContainerDied","Data":"1d2000760c959bce1e4f23044944580b72dd4f7d91283fec96928f16d7833b24"}
Sep 07 22:57:57.077857 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:57.077857    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"
Sep 07 22:57:57.078384 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:57.077880    2582 scope.go:117] "RemoveContainer" containerID="9b961c334dd604573421140e5c7a3a4873f5123b24d5a2fe7cdc01788bac1266"
Sep 07 22:57:57.099511 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:57.099483    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"]
Sep 07 22:57:57.105765 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:57.105735    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-82409b-openshift-default-7d5698f9fb-4wv4k"]
Sep 07 22:57:58.089929 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:57:58.089896    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" path="/var/lib/kubelet/pods/f0c26ae9-651d-471b-b939-6805f3c5f832/volumes"
Sep 07 22:58:01.356900 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:01.356860    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"]
Sep 07 22:58:01.732103 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:01.732006    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"]
Sep 07 22:58:01.732596 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:01.732562    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj" podUID="7115c44c-08ae-428f-8f3c-bfe7342a7d1b" containerName="maas-api" containerID="cri-o://62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819" gracePeriod=30
Sep 07 22:58:01.997778 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:01.997749    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:58:02.105400 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.105366    2582 generic.go:356] "Generic (PLEG): container finished" podID="7115c44c-08ae-428f-8f3c-bfe7342a7d1b" containerID="62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819" exitCode=0
Sep 07 22:58:02.105583 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.105420    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"
Sep 07 22:58:02.105583 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.105449    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj" event={"ID":"7115c44c-08ae-428f-8f3c-bfe7342a7d1b","Type":"ContainerDied","Data":"62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819"}
Sep 07 22:58:02.105583 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.105511    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj" event={"ID":"7115c44c-08ae-428f-8f3c-bfe7342a7d1b","Type":"ContainerDied","Data":"223826d62dc02cc99aae231ad5fedca1d4812c4381ee18826f1b85d3175cd21e"}
Sep 07 22:58:02.105583 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.105535    2582 scope.go:117] "RemoveContainer" containerID="62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819"
Sep 07 22:58:02.116588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.116423    2582 scope.go:117] "RemoveContainer" containerID="62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819"
Sep 07 22:58:02.116920 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:58:02.116865    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819\": container with ID starting with 62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819 not found: ID does not exist" containerID="62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819"
Sep 07 22:58:02.116920 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.116905    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819"} err="failed to get container status \"62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819\": rpc error: code = NotFound desc = could not find container \"62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819\": container with ID starting with 62a390a7eabcaabef5f8534a08eac739dbd33ae0fe3bdda0b63633532ebac819 not found: ID does not exist"
Sep 07 22:58:02.131534 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.131499    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-maas-api-tls\") pod \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") "
Sep 07 22:58:02.131715 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.131573    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6w8t\" (UniqueName: \"kubernetes.io/projected/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-kube-api-access-z6w8t\") pod \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") "
Sep 07 22:58:02.131715 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.131672    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-openshift-service-ca\") pod \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") "
Sep 07 22:58:02.131827 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.131734    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-metrics-certs\") pod \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\" (UID: \"7115c44c-08ae-428f-8f3c-bfe7342a7d1b\") "
Sep 07 22:58:02.132115 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.132080    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7115c44c-08ae-428f-8f3c-bfe7342a7d1b" (UID: "7115c44c-08ae-428f-8f3c-bfe7342a7d1b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 22:58:02.133973 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.133945    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-kube-api-access-z6w8t" (OuterVolumeSpecName: "kube-api-access-z6w8t") pod "7115c44c-08ae-428f-8f3c-bfe7342a7d1b" (UID: "7115c44c-08ae-428f-8f3c-bfe7342a7d1b"). InnerVolumeSpecName "kube-api-access-z6w8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:58:02.134084 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.133990    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7115c44c-08ae-428f-8f3c-bfe7342a7d1b" (UID: "7115c44c-08ae-428f-8f3c-bfe7342a7d1b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:58:02.134084 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.134013    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7115c44c-08ae-428f-8f3c-bfe7342a7d1b" (UID: "7115c44c-08ae-428f-8f3c-bfe7342a7d1b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:58:02.232987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.232890    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.232987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.232926    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.232987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.232936    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.232987 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.232943    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6w8t\" (UniqueName: \"kubernetes.io/projected/7115c44c-08ae-428f-8f3c-bfe7342a7d1b-kube-api-access-z6w8t\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:02.432088 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.432048    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"]
Sep 07 22:58:02.437819 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:02.437786    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-5483b1-556d68d98c-b5ksj"]
Sep 07 22:58:04.091921 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:04.091884    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7115c44c-08ae-428f-8f3c-bfe7342a7d1b" path="/var/lib/kubelet/pods/7115c44c-08ae-428f-8f3c-bfe7342a7d1b/volumes"
Sep 07 22:58:07.134624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.134581    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"]
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.134965    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7115c44c-08ae-428f-8f3c-bfe7342a7d1b" containerName="maas-api"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.134978    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7115c44c-08ae-428f-8f3c-bfe7342a7d1b" containerName="maas-api"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.134990    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.134996    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135010    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135019    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135027    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7d50b72b-1203-43b7-b38d-9170e3e6c824" containerName="payload-pre-processing"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135033    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="7d50b72b-1203-43b7-b38d-9170e3e6c824" containerName="payload-pre-processing"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135044    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5dbab49-36a5-4276-8609-4efb9d722c22" containerName="maas-api"
Sep 07 22:58:07.135047 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135049    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5dbab49-36a5-4276-8609-4efb9d722c22" containerName="maas-api"
Sep 07 22:58:07.135413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135109    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy"
Sep 07 22:58:07.135413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135116    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0c26ae9-651d-471b-b939-6805f3c5f832" containerName="istio-proxy"
Sep 07 22:58:07.135413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135123    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7115c44c-08ae-428f-8f3c-bfe7342a7d1b" containerName="maas-api"
Sep 07 22:58:07.135413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135130    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5dbab49-36a5-4276-8609-4efb9d722c22" containerName="maas-api"
Sep 07 22:58:07.135413 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.135137    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="7d50b72b-1203-43b7-b38d-9170e3e6c824" containerName="payload-pre-processing"
Sep 07 22:58:07.139563 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.139541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.141961 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.141933    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-1dd653\""
Sep 07 22:58:07.142065 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.141973    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-1dd653-dockercfg-nf25j\""
Sep 07 22:58:07.148492 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.148465    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"]
Sep 07 22:58:07.280377 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.280334    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n864n\" (UniqueName: \"kubernetes.io/projected/bb5b4319-6b24-4f52-a252-6d447a4f975d-kube-api-access-n864n\") pod \"payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.280557 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.280450    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5b4319-6b24-4f52-a252-6d447a4f975d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.381336 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.381256    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n864n\" (UniqueName: \"kubernetes.io/projected/bb5b4319-6b24-4f52-a252-6d447a4f975d-kube-api-access-n864n\") pod \"payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.381543 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.381400    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5b4319-6b24-4f52-a252-6d447a4f975d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.382169 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.382141    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5b4319-6b24-4f52-a252-6d447a4f975d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.391497 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.391416    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n864n\" (UniqueName: \"kubernetes.io/projected/bb5b4319-6b24-4f52-a252-6d447a4f975d-kube-api-access-n864n\") pod \"payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.452071 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.452022    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:07.726524 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:07.726492    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"]
Sep 07 22:58:07.728777 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:58:07.728747    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5b4319_6b24_4f52_a252_6d447a4f975d.slice/crio-d5e229a75fa4e2e4347e2476e3ebc913e6807a75c7046cc6cea580327626ee01 WatchSource:0}: Error finding container d5e229a75fa4e2e4347e2476e3ebc913e6807a75c7046cc6cea580327626ee01: Status 404 returned error can't find the container with id d5e229a75fa4e2e4347e2476e3ebc913e6807a75c7046cc6cea580327626ee01
Sep 07 22:58:08.147230 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:08.147194    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7" event={"ID":"bb5b4319-6b24-4f52-a252-6d447a4f975d","Type":"ContainerStarted","Data":"9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997"}
Sep 07 22:58:08.147826 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:08.147250    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7" event={"ID":"bb5b4319-6b24-4f52-a252-6d447a4f975d","Type":"ContainerStarted","Data":"d5e229a75fa4e2e4347e2476e3ebc913e6807a75c7046cc6cea580327626ee01"}
Sep 07 22:58:08.147826 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:08.147330    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:08.162736 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:08.162678    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7" podStartSLOduration=1.16266009 podStartE2EDuration="1.16266009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:08.161143791 +0000 UTC m=+1554.627300787" watchObservedRunningTime="2026-09-07 22:58:08.16266009 +0000 UTC m=+1554.628817085"
Sep 07 22:58:16.384075 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.384008    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8" podUID="2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" containerName="main" containerID="cri-o://9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43" gracePeriod=45
Sep 07 22:58:16.645364 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.645259    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:58:16.673986 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.673948    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cm8cp\" (UniqueName: \"kubernetes.io/projected/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-kube-api-access-cm8cp\") pod \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") "
Sep 07 22:58:16.674176 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674032    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-home\") pod \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") "
Sep 07 22:58:16.674176 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674153    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tmp-dir\") pod \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") "
Sep 07 22:58:16.674325 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674182    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-model-cache\") pod \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") "
Sep 07 22:58:16.674325 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674226    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tls-certs\") pod \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") "
Sep 07 22:58:16.674325 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674253    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-dshm\") pod \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\" (UID: \"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b\") "
Sep 07 22:58:16.674508 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674322    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-home" (OuterVolumeSpecName: "home") pod "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" (UID: "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:16.674508 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674432    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" (UID: "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:16.674605 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674521    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-model-cache" (OuterVolumeSpecName: "model-cache") pod "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" (UID: "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:16.674695 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674671    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-home\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:16.674760 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674698    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tmp-dir\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:16.674760 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.674712    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-model-cache\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:16.676454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.676429    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-dshm" (OuterVolumeSpecName: "dshm") pod "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" (UID: "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 22:58:16.676454 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.676436    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-kube-api-access-cm8cp" (OuterVolumeSpecName: "kube-api-access-cm8cp") pod "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" (UID: "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b"). InnerVolumeSpecName "kube-api-access-cm8cp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 22:58:16.676599 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.676458    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" (UID: "2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 22:58:16.775800 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.775760    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-tls-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:16.775800 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.775794    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-dshm\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:16.775800 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:16.775803    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cm8cp\" (UniqueName: \"kubernetes.io/projected/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b-kube-api-access-cm8cp\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 22:58:17.195351 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.195313    2582 generic.go:356] "Generic (PLEG): container finished" podID="2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" containerID="9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43" exitCode=0
Sep 07 22:58:17.195526 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.195405    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"
Sep 07 22:58:17.195526 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.195418    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8" event={"ID":"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b","Type":"ContainerDied","Data":"9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43"}
Sep 07 22:58:17.195526 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.195451    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8" event={"ID":"2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b","Type":"ContainerDied","Data":"0d3ee835e962ba0742a9cd9b12eb39b08534a8aa3da8ef450467731f5b3138b5"}
Sep 07 22:58:17.195526 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.195467    2582 scope.go:117] "RemoveContainer" containerID="9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43"
Sep 07 22:58:17.205180 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.205155    2582 scope.go:117] "RemoveContainer" containerID="9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43"
Sep 07 22:58:17.205488 ip-10-0-137-80 kubenswrapper[2582]: E0907 22:58:17.205464    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43\": container with ID starting with 9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43 not found: ID does not exist" containerID="9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43"
Sep 07 22:58:17.205566 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.205502    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43"} err="failed to get container status \"9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43\": rpc error: code = NotFound desc = could not find container \"9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43\": container with ID starting with 9a4157bd4d87a8de28f1e9a1d59679c017409097394ee58281bc587df4518c43 not found: ID does not exist"
Sep 07 22:58:17.216591 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.216562    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"]
Sep 07 22:58:17.222188 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:17.222158    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-7823cc/rate-test-model-7823cc-kserve-6965bbdc47-wqtn8"]
Sep 07 22:58:18.090672 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:18.090629    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" path="/var/lib/kubelet/pods/2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b/volumes"
Sep 07 22:58:19.156387 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:19.156349    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 22:58:31.263176 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.263137    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"]
Sep 07 22:58:31.263601 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.263526    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" containerName="main"
Sep 07 22:58:31.263601 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.263540    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" containerName="main"
Sep 07 22:58:31.263687 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.263621    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a7be0ba-7b27-4e29-8b56-98f1a8b70f8b" containerName="main"
Sep 07 22:58:31.268357 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.268329    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.270840 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.270817    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-046f93-openshift-default-dockercfg-ltgjs\""
Sep 07 22:58:31.278211 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.277437    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"]
Sep 07 22:58:31.416467 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416417    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-envoy\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416665 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416478    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-token\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416665 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416544    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-socket\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416665 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416632    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-podinfo\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416823 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416693    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nflqg\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-kube-api-access-nflqg\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416823 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416732    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-data\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416823 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416773    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istiod-ca-cert\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416944 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416838    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-certs\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.416944 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.416903    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-credential-socket\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517758 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517655    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-envoy\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517758 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517716    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-token\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517758    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-socket\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517805    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-podinfo\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517842    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nflqg\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-kube-api-access-nflqg\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517871    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-data\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517911    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istiod-ca-cert\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.517996 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.517954    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-certs\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.518332 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.518012    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-credential-socket\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.518417 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.518388    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-credential-socket\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.518473 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.518453    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-data\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.518641 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.518619    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-socket\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.518730 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.518692    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-certs\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.518912 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.518894    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istiod-ca-cert\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.520352 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.520328    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-envoy\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.520456 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.520374    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-podinfo\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.526411 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.526379    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-token\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.526579 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.526564    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nflqg\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-kube-api-access-nflqg\") pod \"e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") " pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.583588 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.583541    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:31.860251 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.860227    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"]
Sep 07 22:58:31.862720 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:58:31.862692    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf318f321_b5d4_4d45_90b8_6bb9ddfca6e8.slice/crio-19017575d3cbc40222a26dfcfcb80f454a31fc65e3a2a999a1434c4f636ea8f6 WatchSource:0}: Error finding container 19017575d3cbc40222a26dfcfcb80f454a31fc65e3a2a999a1434c4f636ea8f6: Status 404 returned error can't find the container with id 19017575d3cbc40222a26dfcfcb80f454a31fc65e3a2a999a1434c4f636ea8f6
Sep 07 22:58:31.865219 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.865179    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:58:31.865310 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.865267    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:58:31.865364 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:31.865337    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:58:32.278113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:32.278016    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" event={"ID":"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8","Type":"ContainerStarted","Data":"cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19"}
Sep 07 22:58:32.278113 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:32.278067    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" event={"ID":"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8","Type":"ContainerStarted","Data":"19017575d3cbc40222a26dfcfcb80f454a31fc65e3a2a999a1434c4f636ea8f6"}
Sep 07 22:58:32.298821 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:32.298767    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podStartSLOduration=1.29875245 podStartE2EDuration="1.29875245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:32.295683962 +0000 UTC m=+1578.761840963" watchObservedRunningTime="2026-09-07 22:58:32.29875245 +0000 UTC m=+1578.764909446"
Sep 07 22:58:32.584745 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:32.584708    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:32.590366 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:32.590339    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:33.284372 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:33.284340    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:33.285400 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:33.285378    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:35.286461 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.286428    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"]
Sep 07 22:58:35.290098 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.290068    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.292775 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.292734    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-046f93-serving-cert\""
Sep 07 22:58:35.300017 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.299905    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"]
Sep 07 22:58:35.358704 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.358665    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcpl\" (UniqueName: \"kubernetes.io/projected/cb61db41-2660-4c9a-ac1b-8b9e125edb52-kube-api-access-6bcpl\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.358886 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.358735    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb61db41-2660-4c9a-ac1b-8b9e125edb52-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.358886 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.358807    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-maas-api-tls\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.358886 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.358824    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-metrics-certs\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.459927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.459892    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-maas-api-tls\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.459927 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.459929    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-metrics-certs\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.460183 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.459972    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcpl\" (UniqueName: \"kubernetes.io/projected/cb61db41-2660-4c9a-ac1b-8b9e125edb52-kube-api-access-6bcpl\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.460326 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.460298    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb61db41-2660-4c9a-ac1b-8b9e125edb52-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.460902 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.460876    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb61db41-2660-4c9a-ac1b-8b9e125edb52-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.462336 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.462277    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-maas-api-tls\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.462634 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.462610    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-metrics-certs\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.467983 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.467958    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcpl\" (UniqueName: \"kubernetes.io/projected/cb61db41-2660-4c9a-ac1b-8b9e125edb52-kube-api-access-6bcpl\") pod \"maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.604624 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.604582    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:35.886832 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:35.886798    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"]
Sep 07 22:58:35.888429 ip-10-0-137-80 kubenswrapper[2582]: W0907 22:58:35.888403    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb61db41_2660_4c9a_ac1b_8b9e125edb52.slice/crio-a29b0cbe8b3177d3d5d3ef78720d5a077ea0f7e580d7ae176c9a764b18a22bf3 WatchSource:0}: Error finding container a29b0cbe8b3177d3d5d3ef78720d5a077ea0f7e580d7ae176c9a764b18a22bf3: Status 404 returned error can't find the container with id a29b0cbe8b3177d3d5d3ef78720d5a077ea0f7e580d7ae176c9a764b18a22bf3
Sep 07 22:58:36.302789 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:36.302748    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk" event={"ID":"cb61db41-2660-4c9a-ac1b-8b9e125edb52","Type":"ContainerStarted","Data":"2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479"}
Sep 07 22:58:36.303182 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:36.302811    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk" event={"ID":"cb61db41-2660-4c9a-ac1b-8b9e125edb52","Type":"ContainerStarted","Data":"a29b0cbe8b3177d3d5d3ef78720d5a077ea0f7e580d7ae176c9a764b18a22bf3"}
Sep 07 22:58:36.303182 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:36.302851    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:36.318423 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:36.318364    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk" podStartSLOduration=1.034797584 podStartE2EDuration="1.31834571s" totalImagesPullingTime="283.548126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 22:58:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 22:58:36.317863702 +0000 UTC m=+1582.784020696" watchObservedRunningTime="2026-09-07 22:58:36.31834571 +0000 UTC m=+1582.784502705"
Sep 07 22:58:42.313989 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:42.313199    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 22:58:49.326509 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:49.326447    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 07 22:58:49.326998 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:49.326551    2582 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 07 22:58:50.387367 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:50.387331    2582 generic.go:356] "Generic (PLEG): container finished" podID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerID="cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19" exitCode=0
Sep 07 22:58:50.387784 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:50.387403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" event={"ID":"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8","Type":"ContainerDied","Data":"cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19"}
Sep 07 22:58:50.387850 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:50.387835    2582 scope.go:117] "RemoveContainer" containerID="cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19"
Sep 07 22:58:50.389003 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:50.388970    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:58:50.389111 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:50.389044    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:58:50.389111 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:50.389080    2582 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236216Ki","pods":"250"}
Sep 07 22:58:51.394173 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:51.394133    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" event={"ID":"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8","Type":"ContainerStarted","Data":"7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116"}
Sep 07 22:58:51.583982 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:51.583939    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:51.584218 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:51.584192    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 22:58:51.584268 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:51.584247    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 22:58:52.584210 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:52.584165    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 22:58:52.584628 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:52.584223    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 22:58:53.583933 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:53.583896    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 22:58:53.584127 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:53.583950    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 22:58:54.583905 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:54.583871    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 22:58:54.584374 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:54.583933    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 22:58:55.584911 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:55.584869    2582 patch_prober.go:28] interesting pod/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body=
Sep 07 22:58:55.585355 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:55.584946    2582 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused"
Sep 07 22:58:56.687012 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:56.686978    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:56.687525 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:56.687323    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 22:58:56.688276 ip-10-0-137-80 kubenswrapper[2582]: I0907 22:58:56.688257    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 23:00:00.176414 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:00.176375    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"]
Sep 07 23:00:00.180176 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:00.180149    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"
Sep 07 23:00:00.193249 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:00.193216    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"]
Sep 07 23:00:00.196165 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:00.196138    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7fvbm\""
Sep 07 23:00:00.202528 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:00.202494    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"
Sep 07 23:00:00.484564 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:00.484532    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"]
Sep 07 23:00:00.486655 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:00:00.486625    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49bde2e5_bdcd_4b83_aaf6_463fe0472a2d.slice/crio-6a62ce83cdd963ad385c7b0d51d95876e0cd4deacde57af7af916c97ce2a5f60 WatchSource:0}: Error finding container 6a62ce83cdd963ad385c7b0d51d95876e0cd4deacde57af7af916c97ce2a5f60: Status 404 returned error can't find the container with id 6a62ce83cdd963ad385c7b0d51d95876e0cd4deacde57af7af916c97ce2a5f60
Sep 07 23:00:00.781693 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:00.781655    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f" event={"ID":"49bde2e5-bdcd-4b83-aaf6-463fe0472a2d","Type":"ContainerStarted","Data":"6a62ce83cdd963ad385c7b0d51d95876e0cd4deacde57af7af916c97ce2a5f60"}
Sep 07 23:00:01.789231 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:01.789139    2582 generic.go:356] "Generic (PLEG): container finished" podID="49bde2e5-bdcd-4b83-aaf6-463fe0472a2d" containerID="8c791b8b4a1cd29078fc765561348f726ce10a3952f81f22c93fd621bf97323a" exitCode=0
Sep 07 23:00:01.789621 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:01.789226    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f" event={"ID":"49bde2e5-bdcd-4b83-aaf6-463fe0472a2d","Type":"ContainerDied","Data":"8c791b8b4a1cd29078fc765561348f726ce10a3952f81f22c93fd621bf97323a"}
Sep 07 23:00:02.926007 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:02.925980    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"
Sep 07 23:00:03.800630 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:03.800599    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"
Sep 07 23:00:03.800795 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:03.800642    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f" event={"ID":"49bde2e5-bdcd-4b83-aaf6-463fe0472a2d","Type":"ContainerDied","Data":"6a62ce83cdd963ad385c7b0d51d95876e0cd4deacde57af7af916c97ce2a5f60"}
Sep 07 23:00:03.800795 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:03.800693    2582 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6a62ce83cdd963ad385c7b0d51d95876e0cd4deacde57af7af916c97ce2a5f60"
Sep 07 23:00:09.280838 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.280796    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"]
Sep 07 23:00:09.281414 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.281356    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49bde2e5-bdcd-4b83-aaf6-463fe0472a2d" containerName="cleanup"
Sep 07 23:00:09.281414 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.281378    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="49bde2e5-bdcd-4b83-aaf6-463fe0472a2d" containerName="cleanup"
Sep 07 23:00:09.281552 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.281489    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="49bde2e5-bdcd-4b83-aaf6-463fe0472a2d" containerName="cleanup"
Sep 07 23:00:09.284586 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.284561    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.286984 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.286962    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-277d3e-serving-cert\""
Sep 07 23:00:09.294446 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.294414    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"]
Sep 07 23:00:09.389920 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.389876    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.390113 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.390014    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2b6v\" (UniqueName: \"kubernetes.io/projected/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-kube-api-access-v2b6v\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.390113 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.390060    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.390113 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.390083    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.490957 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.490922    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.491161 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.490990    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.491161 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.491112    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2b6v\" (UniqueName: \"kubernetes.io/projected/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-kube-api-access-v2b6v\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.491161 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.491144    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.491772 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.491748    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.493599 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.493577    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.493714 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.493618    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-metrics-certs\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.498590 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.498563    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2b6v\" (UniqueName: \"kubernetes.io/projected/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-kube-api-access-v2b6v\") pod \"maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.597024 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.596979    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:09.873653 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.873568    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"]
Sep 07 23:00:09.877368 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.877349    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:09.879991 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.879972    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-277d3e-dockercfg-b6w76\""
Sep 07 23:00:09.880205 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.880187    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-277d3e\""
Sep 07 23:00:09.881068 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.881043    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"]
Sep 07 23:00:09.883275 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:00:09.883252    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24ef9b3f_e6d0_42d4_9283_c938c22e5fcb.slice/crio-96bea78b6a6c0ceb29ad2909fdb2f51acdf32d7f57caae0c04200bd2b4368af6 WatchSource:0}: Error finding container 96bea78b6a6c0ceb29ad2909fdb2f51acdf32d7f57caae0c04200bd2b4368af6: Status 404 returned error can't find the container with id 96bea78b6a6c0ceb29ad2909fdb2f51acdf32d7f57caae0c04200bd2b4368af6
Sep 07 23:00:09.888024 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.887996    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"]
Sep 07 23:00:09.994977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.994933    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp4hn\" (UniqueName: \"kubernetes.io/projected/b52fce54-3bba-43ce-870f-603d5e8151ab-kube-api-access-cp4hn\") pod \"payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:09.995163 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:09.995062    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52fce54-3bba-43ce-870f-603d5e8151ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:10.096041 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.095998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp4hn\" (UniqueName: \"kubernetes.io/projected/b52fce54-3bba-43ce-870f-603d5e8151ab-kube-api-access-cp4hn\") pod \"payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:10.096243 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.096100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52fce54-3bba-43ce-870f-603d5e8151ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:10.096811 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.096790    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52fce54-3bba-43ce-870f-603d5e8151ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:10.103770 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.103750    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp4hn\" (UniqueName: \"kubernetes.io/projected/b52fce54-3bba-43ce-870f-603d5e8151ab-kube-api-access-cp4hn\") pod \"payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:10.202579 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.202482    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:10.491106 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.491073    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"]
Sep 07 23:00:10.492703 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:00:10.492679    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb52fce54_3bba_43ce_870f_603d5e8151ab.slice/crio-6b957b563b54033032a2e6706d8d169757d5092ef15e652a7968721409030ff0 WatchSource:0}: Error finding container 6b957b563b54033032a2e6706d8d169757d5092ef15e652a7968721409030ff0: Status 404 returned error can't find the container with id 6b957b563b54033032a2e6706d8d169757d5092ef15e652a7968721409030ff0
Sep 07 23:00:10.841183 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.841126    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6" event={"ID":"b52fce54-3bba-43ce-870f-603d5e8151ab","Type":"ContainerStarted","Data":"ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec"}
Sep 07 23:00:10.841425 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.841200    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6" event={"ID":"b52fce54-3bba-43ce-870f-603d5e8151ab","Type":"ContainerStarted","Data":"6b957b563b54033032a2e6706d8d169757d5092ef15e652a7968721409030ff0"}
Sep 07 23:00:10.841425 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.841334    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:10.842896 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.842873    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr" event={"ID":"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb","Type":"ContainerStarted","Data":"0c2b966a3d4616adb87508e51120e3f7ec06a929bc05eefd040f8804902bb269"}
Sep 07 23:00:10.843019 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.842914    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr" event={"ID":"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb","Type":"ContainerStarted","Data":"96bea78b6a6c0ceb29ad2909fdb2f51acdf32d7f57caae0c04200bd2b4368af6"}
Sep 07 23:00:10.843019 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.843006    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:10.858693 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.858641    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6" podStartSLOduration=1.85862521 podStartE2EDuration="1.85862521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:10.856184918 +0000 UTC m=+1677.322341934" watchObservedRunningTime="2026-09-07 23:00:10.85862521 +0000 UTC m=+1677.324782205"
Sep 07 23:00:10.872519 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:10.872466    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr" podStartSLOduration=1.311645817 podStartE2EDuration="1.872448766s" totalImagesPullingTime="560.802949ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:00:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:00:10.870462791 +0000 UTC m=+1677.336619786" watchObservedRunningTime="2026-09-07 23:00:10.872448766 +0000 UTC m=+1677.338605760"
Sep 07 23:00:16.854601 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:16.854568    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:21.852210 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:21.852177    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:39.857533 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:39.857441    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"]
Sep 07 23:00:39.857988 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:39.857753    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr" podUID="24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" containerName="maas-api" containerID="cri-o://0c2b966a3d4616adb87508e51120e3f7ec06a929bc05eefd040f8804902bb269" gracePeriod=30
Sep 07 23:00:39.928477 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:39.928436    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"]
Sep 07 23:00:39.928819 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:39.928709    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6" podUID="b52fce54-3bba-43ce-870f-603d5e8151ab" containerName="payload-pre-processing" containerID="cri-o://ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec" gracePeriod=30
Sep 07 23:00:39.999637 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:39.999603    2582 generic.go:356] "Generic (PLEG): container finished" podID="24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" containerID="0c2b966a3d4616adb87508e51120e3f7ec06a929bc05eefd040f8804902bb269" exitCode=0
Sep 07 23:00:39.999788 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:39.999660    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr" event={"ID":"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb","Type":"ContainerDied","Data":"0c2b966a3d4616adb87508e51120e3f7ec06a929bc05eefd040f8804902bb269"}
Sep 07 23:00:40.188947 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.188919    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:40.214057 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.214025    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:40.307639 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.307607    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-metrics-certs\") pod \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") "
Sep 07 23:00:40.307834 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.307668    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-openshift-service-ca\") pod \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") "
Sep 07 23:00:40.307834 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.307697    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52fce54-3bba-43ce-870f-603d5e8151ab-plugins-config-volume\") pod \"b52fce54-3bba-43ce-870f-603d5e8151ab\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") "
Sep 07 23:00:40.307834 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.307776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-maas-api-tls\") pod \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") "
Sep 07 23:00:40.307834 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.307807    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cp4hn\" (UniqueName: \"kubernetes.io/projected/b52fce54-3bba-43ce-870f-603d5e8151ab-kube-api-access-cp4hn\") pod \"b52fce54-3bba-43ce-870f-603d5e8151ab\" (UID: \"b52fce54-3bba-43ce-870f-603d5e8151ab\") "
Sep 07 23:00:40.308068 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.307846    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-v2b6v\" (UniqueName: \"kubernetes.io/projected/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-kube-api-access-v2b6v\") pod \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\" (UID: \"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb\") "
Sep 07 23:00:40.308158 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.308131    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" (UID: "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:00:40.308220 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.308167    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b52fce54-3bba-43ce-870f-603d5e8151ab-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b52fce54-3bba-43ce-870f-603d5e8151ab" (UID: "b52fce54-3bba-43ce-870f-603d5e8151ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:00:40.309923 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.309894    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b52fce54-3bba-43ce-870f-603d5e8151ab-kube-api-access-cp4hn" (OuterVolumeSpecName: "kube-api-access-cp4hn") pod "b52fce54-3bba-43ce-870f-603d5e8151ab" (UID: "b52fce54-3bba-43ce-870f-603d5e8151ab"). InnerVolumeSpecName "kube-api-access-cp4hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:00:40.310036 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.309924    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" (UID: "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:00:40.310363 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.310335    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-kube-api-access-v2b6v" (OuterVolumeSpecName: "kube-api-access-v2b6v") pod "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" (UID: "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb"). InnerVolumeSpecName "kube-api-access-v2b6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:00:40.310429 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.310356    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" (UID: "24ef9b3f-e6d0-42d4-9283-c938c22e5fcb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:00:40.409393 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.409297    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:00:40.409393 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.409331    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:00:40.409393 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.409343    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b52fce54-3bba-43ce-870f-603d5e8151ab-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:00:40.409393 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.409351    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:00:40.409393 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.409359    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp4hn\" (UniqueName: \"kubernetes.io/projected/b52fce54-3bba-43ce-870f-603d5e8151ab-kube-api-access-cp4hn\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:00:40.409393 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:40.409368    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2b6v\" (UniqueName: \"kubernetes.io/projected/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb-kube-api-access-v2b6v\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:00:41.007264 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.007231    2582 generic.go:356] "Generic (PLEG): container finished" podID="b52fce54-3bba-43ce-870f-603d5e8151ab" containerID="ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec" exitCode=0
Sep 07 23:00:41.007824 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.007297    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"
Sep 07 23:00:41.007824 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.007313    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6" event={"ID":"b52fce54-3bba-43ce-870f-603d5e8151ab","Type":"ContainerDied","Data":"ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec"}
Sep 07 23:00:41.007824 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.007361    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6" event={"ID":"b52fce54-3bba-43ce-870f-603d5e8151ab","Type":"ContainerDied","Data":"6b957b563b54033032a2e6706d8d169757d5092ef15e652a7968721409030ff0"}
Sep 07 23:00:41.007824 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.007382    2582 scope.go:117] "RemoveContainer" containerID="ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec"
Sep 07 23:00:41.008910 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.008890    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr" event={"ID":"24ef9b3f-e6d0-42d4-9283-c938c22e5fcb","Type":"ContainerDied","Data":"96bea78b6a6c0ceb29ad2909fdb2f51acdf32d7f57caae0c04200bd2b4368af6"}
Sep 07 23:00:41.008992 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.008915    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"
Sep 07 23:00:41.021324 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.021225    2582 scope.go:117] "RemoveContainer" containerID="ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec"
Sep 07 23:00:41.022083 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:00:41.021923    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec\": container with ID starting with ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec not found: ID does not exist" containerID="ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec"
Sep 07 23:00:41.022083 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.021962    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec"} err="failed to get container status \"ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec\": rpc error: code = NotFound desc = could not find container \"ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec\": container with ID starting with ebad32267d4aa043668fb1ec2c1d8d42eb75e032c6b0660612aa031bf26892ec not found: ID does not exist"
Sep 07 23:00:41.022083 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.021989    2582 scope.go:117] "RemoveContainer" containerID="0c2b966a3d4616adb87508e51120e3f7ec06a929bc05eefd040f8804902bb269"
Sep 07 23:00:41.034853 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.034824    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"]
Sep 07 23:00:41.040712 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.040685    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-277d3e-86879fd7d-r75hr"]
Sep 07 23:00:41.050354 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.050323    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"]
Sep 07 23:00:41.052646 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:41.052626    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-277d3e-5dfd6f469d-mjrl6"]
Sep 07 23:00:42.090511 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:42.090477    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" path="/var/lib/kubelet/pods/24ef9b3f-e6d0-42d4-9283-c938c22e5fcb/volumes"
Sep 07 23:00:42.090883 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:00:42.090854    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b52fce54-3bba-43ce-870f-603d5e8151ab" path="/var/lib/kubelet/pods/b52fce54-3bba-43ce-870f-603d5e8151ab/volumes"
Sep 07 23:01:05.396634 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.396593    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"]
Sep 07 23:01:05.398791 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.398763    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-1dd653-29813700-9jl5f"]
Sep 07 23:01:05.424101 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.424069    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"]
Sep 07 23:01:05.424419 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.424379    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7" podUID="bb5b4319-6b24-4f52-a252-6d447a4f975d" containerName="payload-pre-processing" containerID="cri-o://9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997" gracePeriod=30
Sep 07 23:01:05.684579 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.684554    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 23:01:05.762339 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.762309    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n864n\" (UniqueName: \"kubernetes.io/projected/bb5b4319-6b24-4f52-a252-6d447a4f975d-kube-api-access-n864n\") pod \"bb5b4319-6b24-4f52-a252-6d447a4f975d\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") "
Sep 07 23:01:05.762339 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.762342    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5b4319-6b24-4f52-a252-6d447a4f975d-plugins-config-volume\") pod \"bb5b4319-6b24-4f52-a252-6d447a4f975d\" (UID: \"bb5b4319-6b24-4f52-a252-6d447a4f975d\") "
Sep 07 23:01:05.762731 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.762708    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5b4319-6b24-4f52-a252-6d447a4f975d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bb5b4319-6b24-4f52-a252-6d447a4f975d" (UID: "bb5b4319-6b24-4f52-a252-6d447a4f975d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:01:05.764509 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.764486    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5b4319-6b24-4f52-a252-6d447a4f975d-kube-api-access-n864n" (OuterVolumeSpecName: "kube-api-access-n864n") pod "bb5b4319-6b24-4f52-a252-6d447a4f975d" (UID: "bb5b4319-6b24-4f52-a252-6d447a4f975d"). InnerVolumeSpecName "kube-api-access-n864n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:01:05.863251 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.863212    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n864n\" (UniqueName: \"kubernetes.io/projected/bb5b4319-6b24-4f52-a252-6d447a4f975d-kube-api-access-n864n\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:01:05.863251 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:05.863245    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bb5b4319-6b24-4f52-a252-6d447a4f975d-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:01:06.090975 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.090932    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49bde2e5-bdcd-4b83-aaf6-463fe0472a2d" path="/var/lib/kubelet/pods/49bde2e5-bdcd-4b83-aaf6-463fe0472a2d/volumes"
Sep 07 23:01:06.153622 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.153587    2582 generic.go:356] "Generic (PLEG): container finished" podID="bb5b4319-6b24-4f52-a252-6d447a4f975d" containerID="9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997" exitCode=0
Sep 07 23:01:06.153800 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.153640    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"
Sep 07 23:01:06.153800 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.153674    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7" event={"ID":"bb5b4319-6b24-4f52-a252-6d447a4f975d","Type":"ContainerDied","Data":"9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997"}
Sep 07 23:01:06.153800 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.153733    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7" event={"ID":"bb5b4319-6b24-4f52-a252-6d447a4f975d","Type":"ContainerDied","Data":"d5e229a75fa4e2e4347e2476e3ebc913e6807a75c7046cc6cea580327626ee01"}
Sep 07 23:01:06.153800 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.153765    2582 scope.go:117] "RemoveContainer" containerID="9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997"
Sep 07 23:01:06.163049 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.163026    2582 scope.go:117] "RemoveContainer" containerID="9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997"
Sep 07 23:01:06.163348 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:01:06.163332    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997\": container with ID starting with 9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997 not found: ID does not exist" containerID="9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997"
Sep 07 23:01:06.163401 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.163358    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997"} err="failed to get container status \"9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997\": rpc error: code = NotFound desc = could not find container \"9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997\": container with ID starting with 9ec67359ae0838e996ee6cc092546b6069b4d31c36377ee7be9f60a02f788997 not found: ID does not exist"
Sep 07 23:01:06.170169 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.170146    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"]
Sep 07 23:01:06.173516 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:06.173492    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-1dd653-ccdbc98c4-p8fk7"]
Sep 07 23:01:08.090162 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:01:08.090130    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb5b4319-6b24-4f52-a252-6d447a4f975d" path="/var/lib/kubelet/pods/bb5b4319-6b24-4f52-a252-6d447a4f975d/volumes"
Sep 07 23:02:09.826204 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:09.826110    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"]
Sep 07 23:02:09.826795 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:09.826406    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk" podUID="cb61db41-2660-4c9a-ac1b-8b9e125edb52" containerName="maas-api" containerID="cri-o://2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479" gracePeriod=30
Sep 07 23:02:10.079167 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.079096    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 23:02:10.180736 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.180695    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb61db41-2660-4c9a-ac1b-8b9e125edb52-openshift-service-ca\") pod \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") "
Sep 07 23:02:10.180736 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.180741    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6bcpl\" (UniqueName: \"kubernetes.io/projected/cb61db41-2660-4c9a-ac1b-8b9e125edb52-kube-api-access-6bcpl\") pod \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") "
Sep 07 23:02:10.180978 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.180776    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-maas-api-tls\") pod \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") "
Sep 07 23:02:10.180978 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.180841    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-metrics-certs\") pod \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\" (UID: \"cb61db41-2660-4c9a-ac1b-8b9e125edb52\") "
Sep 07 23:02:10.181355 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.181318    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb61db41-2660-4c9a-ac1b-8b9e125edb52-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "cb61db41-2660-4c9a-ac1b-8b9e125edb52" (UID: "cb61db41-2660-4c9a-ac1b-8b9e125edb52"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:10.183113 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.183081    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb61db41-2660-4c9a-ac1b-8b9e125edb52-kube-api-access-6bcpl" (OuterVolumeSpecName: "kube-api-access-6bcpl") pod "cb61db41-2660-4c9a-ac1b-8b9e125edb52" (UID: "cb61db41-2660-4c9a-ac1b-8b9e125edb52"). InnerVolumeSpecName "kube-api-access-6bcpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:10.183113 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.183088    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "cb61db41-2660-4c9a-ac1b-8b9e125edb52" (UID: "cb61db41-2660-4c9a-ac1b-8b9e125edb52"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:10.183113 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.183105    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "cb61db41-2660-4c9a-ac1b-8b9e125edb52" (UID: "cb61db41-2660-4c9a-ac1b-8b9e125edb52"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:10.282023 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.281961    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.282023 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.282015    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/cb61db41-2660-4c9a-ac1b-8b9e125edb52-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.282023 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.282029    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bcpl\" (UniqueName: \"kubernetes.io/projected/cb61db41-2660-4c9a-ac1b-8b9e125edb52-kube-api-access-6bcpl\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.282023 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.282041    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/cb61db41-2660-4c9a-ac1b-8b9e125edb52-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:10.512336 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.512231    2582 generic.go:356] "Generic (PLEG): container finished" podID="cb61db41-2660-4c9a-ac1b-8b9e125edb52" containerID="2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479" exitCode=0
Sep 07 23:02:10.512336 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.512273    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"
Sep 07 23:02:10.512532 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.512339    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk" event={"ID":"cb61db41-2660-4c9a-ac1b-8b9e125edb52","Type":"ContainerDied","Data":"2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479"}
Sep 07 23:02:10.512532 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.512394    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk" event={"ID":"cb61db41-2660-4c9a-ac1b-8b9e125edb52","Type":"ContainerDied","Data":"a29b0cbe8b3177d3d5d3ef78720d5a077ea0f7e580d7ae176c9a764b18a22bf3"}
Sep 07 23:02:10.512532 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.512427    2582 scope.go:117] "RemoveContainer" containerID="2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479"
Sep 07 23:02:10.524955 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.524928    2582 scope.go:117] "RemoveContainer" containerID="2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479"
Sep 07 23:02:10.525270 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:02:10.525251    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479\": container with ID starting with 2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479 not found: ID does not exist" containerID="2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479"
Sep 07 23:02:10.525338 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.525294    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479"} err="failed to get container status \"2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479\": rpc error: code = NotFound desc = could not find container \"2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479\": container with ID starting with 2705097f2feb8b839618ea48a376d925e9dd7db03f6ef85f83529209b4e89479 not found: ID does not exist"
Sep 07 23:02:10.538642 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.538613    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"]
Sep 07 23:02:10.541958 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:10.541930    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-046f93-58f76c8f56-hdfjk"]
Sep 07 23:02:12.089564 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:12.089530    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cb61db41-2660-4c9a-ac1b-8b9e125edb52" path="/var/lib/kubelet/pods/cb61db41-2660-4c9a-ac1b-8b9e125edb52/volumes"
Sep 07 23:02:16.121721 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:16.121668    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"]
Sep 07 23:02:16.122739 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:16.121930    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy" containerID="cri-o://7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116" gracePeriod=30
Sep 07 23:02:21.378645 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.378618    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 23:02:21.488673 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488557    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-certs\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488673 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488619    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-socket\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488673 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488643    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-data\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488698    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-credential-socket\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488729    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istiod-ca-cert\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488769    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-envoy\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488815    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nflqg\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-kube-api-access-nflqg\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488854    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-token\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488877    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-podinfo\") pod \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\" (UID: \"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8\") "
Sep 07 23:02:21.488977 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488902    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:21.489387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.488964    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:21.489387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.489047    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:21.489387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.489117    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:21.489387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.489263    2582 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.489387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.489316    2582 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-workload-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.489387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.489330    2582 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-data\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.489387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.489345    2582 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istiod-ca-cert\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.489882 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.489385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:21.491060 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.491030    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-kube-api-access-nflqg" (OuterVolumeSpecName: "kube-api-access-nflqg") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "kube-api-access-nflqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:21.491211 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.491193    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:21.491414 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.491390    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:02:21.491510 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.491444    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" (UID: "f318f321-b5d4-4d45-90b8-6bb9ddfca6e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 07 23:02:21.575034 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.574999    2582 generic.go:356] "Generic (PLEG): container finished" podID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerID="7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116" exitCode=0
Sep 07 23:02:21.575212 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.575052    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"
Sep 07 23:02:21.575212 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.575085    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" event={"ID":"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8","Type":"ContainerDied","Data":"7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116"}
Sep 07 23:02:21.575212 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.575138    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s" event={"ID":"f318f321-b5d4-4d45-90b8-6bb9ddfca6e8","Type":"ContainerDied","Data":"19017575d3cbc40222a26dfcfcb80f454a31fc65e3a2a999a1434c4f636ea8f6"}
Sep 07 23:02:21.575212 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.575159    2582 scope.go:117] "RemoveContainer" containerID="7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116"
Sep 07 23:02:21.584713 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.584693    2582 scope.go:117] "RemoveContainer" containerID="cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19"
Sep 07 23:02:21.590342 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.590320    2582 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-credential-socket\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.590465 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.590343    2582 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-envoy\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.590465 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.590358    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nflqg\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-kube-api-access-nflqg\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.590465 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.590370    2582 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-token\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.590465 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.590383    2582 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8-istio-podinfo\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:21.592825 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.592809    2582 scope.go:117] "RemoveContainer" containerID="7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116"
Sep 07 23:02:21.593141 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:02:21.593123    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116\": container with ID starting with 7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116 not found: ID does not exist" containerID="7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116"
Sep 07 23:02:21.593185 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.593150    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116"} err="failed to get container status \"7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116\": rpc error: code = NotFound desc = could not find container \"7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116\": container with ID starting with 7a965bcf46691a1dcf38306fc8bdfa6bbf19443c597ee047ed2af9da3a194116 not found: ID does not exist"
Sep 07 23:02:21.593185 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.593169    2582 scope.go:117] "RemoveContainer" containerID="cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19"
Sep 07 23:02:21.593579 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:02:21.593552    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19\": container with ID starting with cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19 not found: ID does not exist" containerID="cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19"
Sep 07 23:02:21.593734 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.593710    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19"} err="failed to get container status \"cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19\": rpc error: code = NotFound desc = could not find container \"cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19\": container with ID starting with cd108548970cc49326d9ba0f58cfe1daa2a9e5fdbb7e74bae5aa565475ee5b19 not found: ID does not exist"
Sep 07 23:02:21.596056 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.596031    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"]
Sep 07 23:02:21.601380 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:21.601360    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-046f93-openshift-default-795cd8d86d-5zt5s"]
Sep 07 23:02:22.090298 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:22.090249    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" path="/var/lib/kubelet/pods/f318f321-b5d4-4d45-90b8-6bb9ddfca6e8/volumes"
Sep 07 23:02:26.902266 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:26.902227    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"]
Sep 07 23:02:26.902713 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:26.902539    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt" podUID="ec052a6a-8e39-436b-9608-1ff4fa39dd77" containerName="maas-api" containerID="cri-o://077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5" gracePeriod=30
Sep 07 23:02:26.949976 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:26.949938    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"]
Sep 07 23:02:26.950239 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:26.950208    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk" podUID="e1e88b33-cc4f-416c-9001-c5ea01420519" containerName="payload-processing" containerID="cri-o://935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f" gracePeriod=30
Sep 07 23:02:27.229782 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.229737    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 23:02:27.233424 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.233397    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 23:02:27.240571 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.240541    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg96m\" (UniqueName: \"kubernetes.io/projected/e1e88b33-cc4f-416c-9001-c5ea01420519-kube-api-access-sg96m\") pod \"e1e88b33-cc4f-416c-9001-c5ea01420519\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") "
Sep 07 23:02:27.240711 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.240616    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-metrics-certs\") pod \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") "
Sep 07 23:02:27.240711 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.240701    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/ec052a6a-8e39-436b-9608-1ff4fa39dd77-kube-api-access-npgpl\") pod \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") "
Sep 07 23:02:27.240815 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.240754    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls\") pod \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") "
Sep 07 23:02:27.240815 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.240787    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e88b33-cc4f-416c-9001-c5ea01420519-plugins-config-volume\") pod \"e1e88b33-cc4f-416c-9001-c5ea01420519\" (UID: \"e1e88b33-cc4f-416c-9001-c5ea01420519\") "
Sep 07 23:02:27.240924 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.240825    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec052a6a-8e39-436b-9608-1ff4fa39dd77-openshift-service-ca\") pod \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\" (UID: \"ec052a6a-8e39-436b-9608-1ff4fa39dd77\") "
Sep 07 23:02:27.241203 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.241164    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1e88b33-cc4f-416c-9001-c5ea01420519-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e1e88b33-cc4f-416c-9001-c5ea01420519" (UID: "e1e88b33-cc4f-416c-9001-c5ea01420519"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:27.241499 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.241472    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec052a6a-8e39-436b-9608-1ff4fa39dd77-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "ec052a6a-8e39-436b-9608-1ff4fa39dd77" (UID: "ec052a6a-8e39-436b-9608-1ff4fa39dd77"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:27.242905 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.242880    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1e88b33-cc4f-416c-9001-c5ea01420519-kube-api-access-sg96m" (OuterVolumeSpecName: "kube-api-access-sg96m") pod "e1e88b33-cc4f-416c-9001-c5ea01420519" (UID: "e1e88b33-cc4f-416c-9001-c5ea01420519"). InnerVolumeSpecName "kube-api-access-sg96m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:27.243037 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.242940    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "ec052a6a-8e39-436b-9608-1ff4fa39dd77" (UID: "ec052a6a-8e39-436b-9608-1ff4fa39dd77"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:27.243217 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.243201    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "ec052a6a-8e39-436b-9608-1ff4fa39dd77" (UID: "ec052a6a-8e39-436b-9608-1ff4fa39dd77"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:27.243325 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.243222    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec052a6a-8e39-436b-9608-1ff4fa39dd77-kube-api-access-npgpl" (OuterVolumeSpecName: "kube-api-access-npgpl") pod "ec052a6a-8e39-436b-9608-1ff4fa39dd77" (UID: "ec052a6a-8e39-436b-9608-1ff4fa39dd77"). InnerVolumeSpecName "kube-api-access-npgpl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:27.342044 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.341998    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-npgpl\" (UniqueName: \"kubernetes.io/projected/ec052a6a-8e39-436b-9608-1ff4fa39dd77-kube-api-access-npgpl\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.342044 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.342038    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.342044 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.342048    2582 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e1e88b33-cc4f-416c-9001-c5ea01420519-plugins-config-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.342044 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.342057    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ec052a6a-8e39-436b-9608-1ff4fa39dd77-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.342373 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.342067    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg96m\" (UniqueName: \"kubernetes.io/projected/e1e88b33-cc4f-416c-9001-c5ea01420519-kube-api-access-sg96m\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.342373 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.342075    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ec052a6a-8e39-436b-9608-1ff4fa39dd77-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:27.609877 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.609840    2582 generic.go:356] "Generic (PLEG): container finished" podID="ec052a6a-8e39-436b-9608-1ff4fa39dd77" containerID="077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5" exitCode=0
Sep 07 23:02:27.610073 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.609885    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"
Sep 07 23:02:27.610073 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.609926    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt" event={"ID":"ec052a6a-8e39-436b-9608-1ff4fa39dd77","Type":"ContainerDied","Data":"077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5"}
Sep 07 23:02:27.610073 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.609993    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt" event={"ID":"ec052a6a-8e39-436b-9608-1ff4fa39dd77","Type":"ContainerDied","Data":"4dc17dc7ab797ef1b5b33b497a0b96df8ca405030a5162fae7d7c749a0fe0887"}
Sep 07 23:02:27.610073 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.610029    2582 scope.go:117] "RemoveContainer" containerID="077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5"
Sep 07 23:02:27.611327 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.611273    2582 generic.go:356] "Generic (PLEG): container finished" podID="e1e88b33-cc4f-416c-9001-c5ea01420519" containerID="935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f" exitCode=0
Sep 07 23:02:27.611453 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.611339    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"
Sep 07 23:02:27.611453 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.611365    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk" event={"ID":"e1e88b33-cc4f-416c-9001-c5ea01420519","Type":"ContainerDied","Data":"935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f"}
Sep 07 23:02:27.611453 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.611403    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk" event={"ID":"e1e88b33-cc4f-416c-9001-c5ea01420519","Type":"ContainerDied","Data":"fb56ca17334aea4a9fc95f1a546be43c9dc88fecfdd8de1f36bd61713b7470a9"}
Sep 07 23:02:27.621841 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.621701    2582 scope.go:117] "RemoveContainer" containerID="077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5"
Sep 07 23:02:27.622091 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:02:27.622060    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5\": container with ID starting with 077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5 not found: ID does not exist" containerID="077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5"
Sep 07 23:02:27.622190 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.622096    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5"} err="failed to get container status \"077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5\": rpc error: code = NotFound desc = could not find container \"077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5\": container with ID starting with 077c555dd987df70b8d2ee5bdda50a5a80c326208fc18d9dd9e734b11b8eb9b5 not found: ID does not exist"
Sep 07 23:02:27.622190 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.622121    2582 scope.go:117] "RemoveContainer" containerID="935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f"
Sep 07 23:02:27.630202 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.630185    2582 scope.go:117] "RemoveContainer" containerID="935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f"
Sep 07 23:02:27.630903 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:02:27.630882    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f\": container with ID starting with 935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f not found: ID does not exist" containerID="935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f"
Sep 07 23:02:27.630960 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.630911    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f"} err="failed to get container status \"935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f\": rpc error: code = NotFound desc = could not find container \"935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f\": container with ID starting with 935dafad605bb3264bac22281064c1537c4c17939561887fd33109f2998d0e1f not found: ID does not exist"
Sep 07 23:02:27.636709 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.636685    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"]
Sep 07 23:02:27.640228 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.640209    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w3-1d0395-c4dfcdbf8-gz9wt"]
Sep 07 23:02:27.653958 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.653931    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"]
Sep 07 23:02:27.657499 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:27.657480    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-1d0395-7ffd97c97f-jjmzk"]
Sep 07 23:02:28.091239 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:28.091198    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1e88b33-cc4f-416c-9001-c5ea01420519" path="/var/lib/kubelet/pods/e1e88b33-cc4f-416c-9001-c5ea01420519/volumes"
Sep 07 23:02:28.091659 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:28.091643    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ec052a6a-8e39-436b-9608-1ff4fa39dd77" path="/var/lib/kubelet/pods/ec052a6a-8e39-436b-9608-1ff4fa39dd77/volumes"
Sep 07 23:02:44.293091 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.293047    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"]
Sep 07 23:02:44.293729 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.293326    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6" podUID="3339f274-c827-4374-8613-963e27cd8340" containerName="maas-api" containerID="cri-o://3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f" gracePeriod=30
Sep 07 23:02:44.545151 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.545084    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 23:02:44.603698 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.603655    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-metrics-certs\") pod \"3339f274-c827-4374-8613-963e27cd8340\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") "
Sep 07 23:02:44.603914 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.603731    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-maas-api-tls\") pod \"3339f274-c827-4374-8613-963e27cd8340\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") "
Sep 07 23:02:44.603914 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.603806    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w6crf\" (UniqueName: \"kubernetes.io/projected/3339f274-c827-4374-8613-963e27cd8340-kube-api-access-w6crf\") pod \"3339f274-c827-4374-8613-963e27cd8340\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") "
Sep 07 23:02:44.603914 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.603832    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3339f274-c827-4374-8613-963e27cd8340-openshift-service-ca\") pod \"3339f274-c827-4374-8613-963e27cd8340\" (UID: \"3339f274-c827-4374-8613-963e27cd8340\") "
Sep 07 23:02:44.604277 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.604248    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3339f274-c827-4374-8613-963e27cd8340-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3339f274-c827-4374-8613-963e27cd8340" (UID: "3339f274-c827-4374-8613-963e27cd8340"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:02:44.605924 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.605886    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3339f274-c827-4374-8613-963e27cd8340" (UID: "3339f274-c827-4374-8613-963e27cd8340"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:44.605924 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.605904    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "3339f274-c827-4374-8613-963e27cd8340" (UID: "3339f274-c827-4374-8613-963e27cd8340"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:02:44.606062 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.605929    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3339f274-c827-4374-8613-963e27cd8340-kube-api-access-w6crf" (OuterVolumeSpecName: "kube-api-access-w6crf") pod "3339f274-c827-4374-8613-963e27cd8340" (UID: "3339f274-c827-4374-8613-963e27cd8340"). InnerVolumeSpecName "kube-api-access-w6crf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:02:44.701328 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.701277    2582 generic.go:356] "Generic (PLEG): container finished" podID="3339f274-c827-4374-8613-963e27cd8340" containerID="3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f" exitCode=0
Sep 07 23:02:44.701522 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.701340    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"
Sep 07 23:02:44.701522 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.701385    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6" event={"ID":"3339f274-c827-4374-8613-963e27cd8340","Type":"ContainerDied","Data":"3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f"}
Sep 07 23:02:44.701522 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.701441    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6" event={"ID":"3339f274-c827-4374-8613-963e27cd8340","Type":"ContainerDied","Data":"145e50d20879892cc826a75b89b0958fae81ec74f434b57e28268cba89356a05"}
Sep 07 23:02:44.701522 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.701465    2582 scope.go:117] "RemoveContainer" containerID="3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f"
Sep 07 23:02:44.704928 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.704891    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:44.704928 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.704928    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6crf\" (UniqueName: \"kubernetes.io/projected/3339f274-c827-4374-8613-963e27cd8340-kube-api-access-w6crf\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:44.705098 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.704944    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3339f274-c827-4374-8613-963e27cd8340-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:44.705098 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.704958    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3339f274-c827-4374-8613-963e27cd8340-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:02:44.711708 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.711689    2582 scope.go:117] "RemoveContainer" containerID="3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f"
Sep 07 23:02:44.712052 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:02:44.712030    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f\": container with ID starting with 3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f not found: ID does not exist" containerID="3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f"
Sep 07 23:02:44.712129 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.712060    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f"} err="failed to get container status \"3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f\": rpc error: code = NotFound desc = could not find container \"3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f\": container with ID starting with 3c5a3b7b076dbb15d503cb07e10639d6b8cb3004e577544cf7718c5f3a14d78f not found: ID does not exist"
Sep 07 23:02:44.723017 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.722976    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"]
Sep 07 23:02:44.725978 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:44.725949    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w3-7823cc-c8756fbc7-8cnp6"]
Sep 07 23:02:46.089168 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:02:46.089135    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3339f274-c827-4374-8613-963e27cd8340" path="/var/lib/kubelet/pods/3339f274-c827-4374-8613-963e27cd8340/volumes"
Sep 07 23:05:22.799067 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:22.799029    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"]
Sep 07 23:05:22.799623 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:22.799247    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z" podUID="b05dc44d-ac3e-432a-93d1-67dad46fc24b" containerName="manager" containerID="cri-o://9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35" gracePeriod=10
Sep 07 23:05:25.448475 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.448444    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 23:05:25.475221 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.475175    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05dc44d-ac3e-432a-93d1-67dad46fc24b-extensions-socket-volume\") pod \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") "
Sep 07 23:05:25.475418 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.475265    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtjdr\" (UniqueName: \"kubernetes.io/projected/b05dc44d-ac3e-432a-93d1-67dad46fc24b-kube-api-access-vtjdr\") pod \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\" (UID: \"b05dc44d-ac3e-432a-93d1-67dad46fc24b\") "
Sep 07 23:05:25.475666 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.475639    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b05dc44d-ac3e-432a-93d1-67dad46fc24b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b05dc44d-ac3e-432a-93d1-67dad46fc24b" (UID: "b05dc44d-ac3e-432a-93d1-67dad46fc24b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:05:25.477586 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.477554    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b05dc44d-ac3e-432a-93d1-67dad46fc24b-kube-api-access-vtjdr" (OuterVolumeSpecName: "kube-api-access-vtjdr") pod "b05dc44d-ac3e-432a-93d1-67dad46fc24b" (UID: "b05dc44d-ac3e-432a-93d1-67dad46fc24b"). InnerVolumeSpecName "kube-api-access-vtjdr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:05:25.514894 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.514859    2582 generic.go:356] "Generic (PLEG): container finished" podID="b05dc44d-ac3e-432a-93d1-67dad46fc24b" containerID="9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35" exitCode=0
Sep 07 23:05:25.515064 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.514906    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"
Sep 07 23:05:25.515064 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.514953    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z" event={"ID":"b05dc44d-ac3e-432a-93d1-67dad46fc24b","Type":"ContainerDied","Data":"9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35"}
Sep 07 23:05:25.515064 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.515046    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z" event={"ID":"b05dc44d-ac3e-432a-93d1-67dad46fc24b","Type":"ContainerDied","Data":"d4dfa0b0386d3a638b975d33541ca0ae85e1766d413f0c33f1b35c61d88589f2"}
Sep 07 23:05:25.515173 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.515076    2582 scope.go:117] "RemoveContainer" containerID="9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35"
Sep 07 23:05:25.525841 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.525821    2582 scope.go:117] "RemoveContainer" containerID="9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35"
Sep 07 23:05:25.526142 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:05:25.526120    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35\": container with ID starting with 9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35 not found: ID does not exist" containerID="9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35"
Sep 07 23:05:25.526255 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.526149    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35"} err="failed to get container status \"9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35\": rpc error: code = NotFound desc = could not find container \"9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35\": container with ID starting with 9b2815c28082dc515f38b4478cb54a02f6f6b756283dda9743585787a9aa4a35 not found: ID does not exist"
Sep 07 23:05:25.536895 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.536863    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"]
Sep 07 23:05:25.539969 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.539941    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-xrq2z"]
Sep 07 23:05:25.576208 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.576133    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtjdr\" (UniqueName: \"kubernetes.io/projected/b05dc44d-ac3e-432a-93d1-67dad46fc24b-kube-api-access-vtjdr\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:05:25.576208 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:25.576159    2582 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b05dc44d-ac3e-432a-93d1-67dad46fc24b-extensions-socket-volume\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:05:26.089572 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:05:26.089536    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b05dc44d-ac3e-432a-93d1-67dad46fc24b" path="/var/lib/kubelet/pods/b05dc44d-ac3e-432a-93d1-67dad46fc24b/volumes"
Sep 07 23:06:14.354231 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:14.354201    2582 scope.go:117] "RemoveContainer" containerID="8c791b8b4a1cd29078fc765561348f726ce10a3952f81f22c93fd621bf97323a"
Sep 07 23:06:23.466179 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466141    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"]
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466469    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466482    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466492    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b05dc44d-ac3e-432a-93d1-67dad46fc24b" containerName="manager"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466499    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b05dc44d-ac3e-432a-93d1-67dad46fc24b" containerName="manager"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466511    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb5b4319-6b24-4f52-a252-6d447a4f975d" containerName="payload-pre-processing"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466518    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb5b4319-6b24-4f52-a252-6d447a4f975d" containerName="payload-pre-processing"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466526    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3339f274-c827-4374-8613-963e27cd8340" containerName="maas-api"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466532    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="3339f274-c827-4374-8613-963e27cd8340" containerName="maas-api"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466538    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466543    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466554    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1e88b33-cc4f-416c-9001-c5ea01420519" containerName="payload-processing"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466560    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1e88b33-cc4f-416c-9001-c5ea01420519" containerName="payload-processing"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466566    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ec052a6a-8e39-436b-9608-1ff4fa39dd77" containerName="maas-api"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466571    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="ec052a6a-8e39-436b-9608-1ff4fa39dd77" containerName="maas-api"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466576    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cb61db41-2660-4c9a-ac1b-8b9e125edb52" containerName="maas-api"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466581    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="cb61db41-2660-4c9a-ac1b-8b9e125edb52" containerName="maas-api"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466591    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" containerName="maas-api"
Sep 07 23:06:23.466588 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466596    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" containerName="maas-api"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466601    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b52fce54-3bba-43ce-870f-603d5e8151ab" containerName="payload-pre-processing"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466606    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b52fce54-3bba-43ce-870f-603d5e8151ab" containerName="payload-pre-processing"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466655    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="3339f274-c827-4374-8613-963e27cd8340" containerName="maas-api"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466662    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b52fce54-3bba-43ce-870f-603d5e8151ab" containerName="payload-pre-processing"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466667    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466673    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="f318f321-b5d4-4d45-90b8-6bb9ddfca6e8" containerName="istio-proxy"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466678    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1e88b33-cc4f-416c-9001-c5ea01420519" containerName="payload-processing"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466682    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb5b4319-6b24-4f52-a252-6d447a4f975d" containerName="payload-pre-processing"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466687    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="24ef9b3f-e6d0-42d4-9283-c938c22e5fcb" containerName="maas-api"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466691    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="ec052a6a-8e39-436b-9608-1ff4fa39dd77" containerName="maas-api"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466698    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b05dc44d-ac3e-432a-93d1-67dad46fc24b" containerName="manager"
Sep 07 23:06:23.467138 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.466703    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="cb61db41-2660-4c9a-ac1b-8b9e125edb52" containerName="maas-api"
Sep 07 23:06:23.469504 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.469488    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:23.471887 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.471862    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-cczlv\""
Sep 07 23:06:23.483772 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.483738    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"]
Sep 07 23:06:23.596446 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.596406    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nbrb\" (UniqueName: \"kubernetes.io/projected/31c3211e-37c5-46ec-977e-18b69388eb7d-kube-api-access-4nbrb\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w6glk\" (UID: \"31c3211e-37c5-46ec-977e-18b69388eb7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:23.596639 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.596526    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c3211e-37c5-46ec-977e-18b69388eb7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w6glk\" (UID: \"31c3211e-37c5-46ec-977e-18b69388eb7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:23.697194 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.697155    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nbrb\" (UniqueName: \"kubernetes.io/projected/31c3211e-37c5-46ec-977e-18b69388eb7d-kube-api-access-4nbrb\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w6glk\" (UID: \"31c3211e-37c5-46ec-977e-18b69388eb7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:23.697350 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.697325    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c3211e-37c5-46ec-977e-18b69388eb7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w6glk\" (UID: \"31c3211e-37c5-46ec-977e-18b69388eb7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:23.697689 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.697670    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/31c3211e-37c5-46ec-977e-18b69388eb7d-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w6glk\" (UID: \"31c3211e-37c5-46ec-977e-18b69388eb7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:23.706260 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.706230    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nbrb\" (UniqueName: \"kubernetes.io/projected/31c3211e-37c5-46ec-977e-18b69388eb7d-kube-api-access-4nbrb\") pod \"kuadrant-operator-controller-manager-5b65f94d4c-w6glk\" (UID: \"31c3211e-37c5-46ec-977e-18b69388eb7d\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:23.779186 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:23.779101    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:24.052880 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:24.052843    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"]
Sep 07 23:06:24.058739 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:06:24.058699    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31c3211e_37c5_46ec_977e_18b69388eb7d.slice/crio-9a0bdd2b49b7620d9c51c786424fecb349b3113531b522021cf34599c77b70c3 WatchSource:0}: Error finding container 9a0bdd2b49b7620d9c51c786424fecb349b3113531b522021cf34599c77b70c3: Status 404 returned error can't find the container with id 9a0bdd2b49b7620d9c51c786424fecb349b3113531b522021cf34599c77b70c3
Sep 07 23:06:24.814187 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:24.814117    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk" event={"ID":"31c3211e-37c5-46ec-977e-18b69388eb7d","Type":"ContainerStarted","Data":"7558cab2f7dbaaaf376a9239fbce307f0d7741bf4546755bf3c86d4a4e20c7f2"}
Sep 07 23:06:24.814187 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:24.814191    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:06:24.814750 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:24.814219    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk" event={"ID":"31c3211e-37c5-46ec-977e-18b69388eb7d","Type":"ContainerStarted","Data":"9a0bdd2b49b7620d9c51c786424fecb349b3113531b522021cf34599c77b70c3"}
Sep 07 23:06:24.836853 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:24.836804    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk" podStartSLOduration=1.836788663 podStartE2EDuration="1.836788663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:06:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:06:24.834778277 +0000 UTC m=+2051.300935271" watchObservedRunningTime="2026-09-07 23:06:24.836788663 +0000 UTC m=+2051.302945657"
Sep 07 23:06:35.821438 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:06:35.821403    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5b65f94d4c-w6glk"
Sep 07 23:07:20.462000 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.461967    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"]
Sep 07 23:07:20.465480 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.465454    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.467717 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.467694    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-bc653fb1-kserve-self-signed-certs\""
Sep 07 23:07:20.475768 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.475743    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"]
Sep 07 23:07:20.622381 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.622341    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tls-certs\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.622558 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.622395    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7qnn\" (UniqueName: \"kubernetes.io/projected/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-kube-api-access-j7qnn\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.622558 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.622431    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tmp-dir\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.622634 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.622572    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-home\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.622634 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.622620    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-model-cache\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.622699 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.622669    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-dshm\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724129 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724033    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-model-cache\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724129 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724100    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-dshm\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724409 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724147    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tls-certs\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724409 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724195    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7qnn\" (UniqueName: \"kubernetes.io/projected/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-kube-api-access-j7qnn\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724409 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724327    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tmp-dir\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724595 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724437    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-model-cache\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724595 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724450    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-home\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724726 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724704    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tmp-dir\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.724825 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.724802    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-home\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.726552 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.726527    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-dshm\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.727572 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.727543    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tls-certs\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.732271 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.732237    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7qnn\" (UniqueName: \"kubernetes.io/projected/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-kube-api-access-j7qnn\") pod \"e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") " pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:20.776114 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:20.776078    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:21.050033 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:21.050007    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"]
Sep 07 23:07:21.052240 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:07:21.052208    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc6c9873b_302d_4c3c_818a_ab5a68af6ba2.slice/crio-d9b769be8f5f59d51bf8ac5b694616a5121b0a146405855d84a2a6536ca844b8 WatchSource:0}: Error finding container d9b769be8f5f59d51bf8ac5b694616a5121b0a146405855d84a2a6536ca844b8: Status 404 returned error can't find the container with id d9b769be8f5f59d51bf8ac5b694616a5121b0a146405855d84a2a6536ca844b8
Sep 07 23:07:21.107094 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:21.107061    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6" event={"ID":"c6c9873b-302d-4c3c-818a-ab5a68af6ba2","Type":"ContainerStarted","Data":"d9b769be8f5f59d51bf8ac5b694616a5121b0a146405855d84a2a6536ca844b8"}
Sep 07 23:07:22.120210 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:22.120163    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6" event={"ID":"c6c9873b-302d-4c3c-818a-ab5a68af6ba2","Type":"ContainerStarted","Data":"e1db219cd18e1805eef71c009eb0888e40d80f2cf4725cd0df6d9365774bc012"}
Sep 07 23:07:22.137214 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:22.137136    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6" podStartSLOduration=2.13711914 podStartE2EDuration="2.13711914s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:07:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:07:22.134796049 +0000 UTC m=+2108.600953046" watchObservedRunningTime="2026-09-07 23:07:22.13711914 +0000 UTC m=+2108.603276134"
Sep 07 23:07:30.776767 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:30.776731    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:30.777357 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:30.776780    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:30.789379 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:30.789350    2582 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:31.176254 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:31.176221    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:07:48.090864 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:07:48.090832    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"]
Sep 07 23:08:03.110696 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.110647    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6" podUID="c6c9873b-302d-4c3c-818a-ab5a68af6ba2" containerName="main" containerID="cri-o://e1db219cd18e1805eef71c009eb0888e40d80f2cf4725cd0df6d9365774bc012" gracePeriod=45
Sep 07 23:08:03.328640 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.328602    2582 generic.go:356] "Generic (PLEG): container finished" podID="c6c9873b-302d-4c3c-818a-ab5a68af6ba2" containerID="e1db219cd18e1805eef71c009eb0888e40d80f2cf4725cd0df6d9365774bc012" exitCode=0
Sep 07 23:08:03.328811 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.328680    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6" event={"ID":"c6c9873b-302d-4c3c-818a-ab5a68af6ba2","Type":"ContainerDied","Data":"e1db219cd18e1805eef71c009eb0888e40d80f2cf4725cd0df6d9365774bc012"}
Sep 07 23:08:03.363969 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.363899    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:08:03.534874 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.534836    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-dshm\") pod \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") "
Sep 07 23:08:03.535075 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.534884    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-home\") pod \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") "
Sep 07 23:08:03.535075 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.534969    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tls-certs\") pod \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") "
Sep 07 23:08:03.535075 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535020    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tmp-dir\") pod \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") "
Sep 07 23:08:03.535075 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535057    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j7qnn\" (UniqueName: \"kubernetes.io/projected/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-kube-api-access-j7qnn\") pod \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") "
Sep 07 23:08:03.535325 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535082    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-model-cache\") pod \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\" (UID: \"c6c9873b-302d-4c3c-818a-ab5a68af6ba2\") "
Sep 07 23:08:03.535325 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535260    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-home" (OuterVolumeSpecName: "home") pod "c6c9873b-302d-4c3c-818a-ab5a68af6ba2" (UID: "c6c9873b-302d-4c3c-818a-ab5a68af6ba2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:03.535413 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535341    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c6c9873b-302d-4c3c-818a-ab5a68af6ba2" (UID: "c6c9873b-302d-4c3c-818a-ab5a68af6ba2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:03.535463 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535444    2582 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tmp-dir\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:03.535500 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535464    2582 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-home\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:03.535546 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.535492    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-model-cache" (OuterVolumeSpecName: "model-cache") pod "c6c9873b-302d-4c3c-818a-ab5a68af6ba2" (UID: "c6c9873b-302d-4c3c-818a-ab5a68af6ba2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:03.537367 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.537336    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-dshm" (OuterVolumeSpecName: "dshm") pod "c6c9873b-302d-4c3c-818a-ab5a68af6ba2" (UID: "c6c9873b-302d-4c3c-818a-ab5a68af6ba2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 07 23:08:03.537508 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.537385    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c6c9873b-302d-4c3c-818a-ab5a68af6ba2" (UID: "c6c9873b-302d-4c3c-818a-ab5a68af6ba2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:08:03.537575 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.537518    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-kube-api-access-j7qnn" (OuterVolumeSpecName: "kube-api-access-j7qnn") pod "c6c9873b-302d-4c3c-818a-ab5a68af6ba2" (UID: "c6c9873b-302d-4c3c-818a-ab5a68af6ba2"). InnerVolumeSpecName "kube-api-access-j7qnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:08:03.636223 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.636130    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7qnn\" (UniqueName: \"kubernetes.io/projected/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-kube-api-access-j7qnn\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:03.636223 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.636163    2582 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-model-cache\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:03.636223 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.636174    2582 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-dshm\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:03.636223 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:03.636184    2582 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6c9873b-302d-4c3c-818a-ab5a68af6ba2-tls-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:04.335084 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.335051    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"
Sep 07 23:08:04.335084 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.335057    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6" event={"ID":"c6c9873b-302d-4c3c-818a-ab5a68af6ba2","Type":"ContainerDied","Data":"d9b769be8f5f59d51bf8ac5b694616a5121b0a146405855d84a2a6536ca844b8"}
Sep 07 23:08:04.335593 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.335114    2582 scope.go:117] "RemoveContainer" containerID="e1db219cd18e1805eef71c009eb0888e40d80f2cf4725cd0df6d9365774bc012"
Sep 07 23:08:04.347023 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.346951    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"]
Sep 07 23:08:04.347882 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.347857    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6c9873b-302d-4c3c-818a-ab5a68af6ba2" containerName="main"
Sep 07 23:08:04.348038 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.348025    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6c9873b-302d-4c3c-818a-ab5a68af6ba2" containerName="main"
Sep 07 23:08:04.348351 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.348337    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6c9873b-302d-4c3c-818a-ab5a68af6ba2" containerName="main"
Sep 07 23:08:04.353628 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.353598    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.363567 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.363543    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 07 23:08:04.363665 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.363615    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 07 23:08:04.363665 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.363622    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 07 23:08:04.363964 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.363949    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-8mcs4\""
Sep 07 23:08:04.364432 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.364419    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 07 23:08:04.369731 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.369710    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"]
Sep 07 23:08:04.386120 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.386091    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"]
Sep 07 23:08:04.398673 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.398634    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-bc653fb1-kserve-7b556c6bf5-bx2n6"]
Sep 07 23:08:04.545112 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.545073    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-metrics-certs\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.545309 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.545122    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-maas-api-tls\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.545309 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.545174    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pps7t\" (UniqueName: \"kubernetes.io/projected/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-kube-api-access-pps7t\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.545309 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.545212    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-openshift-service-ca\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.646144 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.646054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-maas-api-tls\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.646144 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.646109    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pps7t\" (UniqueName: \"kubernetes.io/projected/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-kube-api-access-pps7t\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.646373 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.646165    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-openshift-service-ca\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.646373 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.646241    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-metrics-certs\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.646945 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.646923    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-openshift-service-ca\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.648568 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.648541    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-maas-api-tls\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.648681 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.648615    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-metrics-certs\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.653647 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.653620    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pps7t\" (UniqueName: \"kubernetes.io/projected/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-kube-api-access-pps7t\") pod \"maas-api-58d7778d67-dcq9b\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") " pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.663520 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.663498    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:04.949667 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.949634    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"]
Sep 07 23:08:04.952375 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:08:04.952340    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb32ea046_2283_48ee_a7a6_b2b42cbc92a5.slice/crio-c134e0b7aeb5ffe74e3656df13deea9bfcb2614c297d950e36b90b43830b83fd WatchSource:0}: Error finding container c134e0b7aeb5ffe74e3656df13deea9bfcb2614c297d950e36b90b43830b83fd: Status 404 returned error can't find the container with id c134e0b7aeb5ffe74e3656df13deea9bfcb2614c297d950e36b90b43830b83fd
Sep 07 23:08:04.954400 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:04.954376    2582 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 07 23:08:05.341882 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:05.341838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b" event={"ID":"b32ea046-2283-48ee-a7a6-b2b42cbc92a5","Type":"ContainerStarted","Data":"c134e0b7aeb5ffe74e3656df13deea9bfcb2614c297d950e36b90b43830b83fd"}
Sep 07 23:08:06.089978 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:06.089876    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6c9873b-302d-4c3c-818a-ab5a68af6ba2" path="/var/lib/kubelet/pods/c6c9873b-302d-4c3c-818a-ab5a68af6ba2/volumes"
Sep 07 23:08:06.349531 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:06.349422    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b" event={"ID":"b32ea046-2283-48ee-a7a6-b2b42cbc92a5","Type":"ContainerStarted","Data":"402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1"}
Sep 07 23:08:06.349531 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:06.349489    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:06.364669 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:06.364617    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b" podStartSLOduration=1.9466973319999998 podStartE2EDuration="2.364603327s" totalImagesPullingTime="417.905995ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:08:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:08:06.363971512 +0000 UTC m=+2152.830128510" watchObservedRunningTime="2026-09-07 23:08:06.364603327 +0000 UTC m=+2152.830760322"
Sep 07 23:08:11.679810 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.679777    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"]
Sep 07 23:08:11.680405 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.680010    2582 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b" podUID="b32ea046-2283-48ee-a7a6-b2b42cbc92a5" containerName="maas-api" containerID="cri-o://402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1" gracePeriod=30
Sep 07 23:08:11.685677 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.685654    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:11.711193 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.711168    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"]
Sep 07 23:08:11.714748 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.714428    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.723113 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.723083    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"]
Sep 07 23:08:11.806396 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.806365    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqkq8\" (UniqueName: \"kubernetes.io/projected/3d7f6513-e1d8-4314-a352-478d97d59288-kube-api-access-jqkq8\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.806549 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.806403    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f6513-e1d8-4314-a352-478d97d59288-openshift-service-ca\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.806549 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.806483    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d7f6513-e1d8-4314-a352-478d97d59288-maas-api-tls\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.806549 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.806510    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7f6513-e1d8-4314-a352-478d97d59288-metrics-certs\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.906988 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.906955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqkq8\" (UniqueName: \"kubernetes.io/projected/3d7f6513-e1d8-4314-a352-478d97d59288-kube-api-access-jqkq8\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.906988 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.906998    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f6513-e1d8-4314-a352-478d97d59288-openshift-service-ca\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.907197 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.907054    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d7f6513-e1d8-4314-a352-478d97d59288-maas-api-tls\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.907197 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.907070    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7f6513-e1d8-4314-a352-478d97d59288-metrics-certs\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.907810 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.907782    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d7f6513-e1d8-4314-a352-478d97d59288-openshift-service-ca\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.909549 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.909520    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3d7f6513-e1d8-4314-a352-478d97d59288-metrics-certs\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.909658 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.909569    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3d7f6513-e1d8-4314-a352-478d97d59288-maas-api-tls\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.914632 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.914607    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqkq8\" (UniqueName: \"kubernetes.io/projected/3d7f6513-e1d8-4314-a352-478d97d59288-kube-api-access-jqkq8\") pod \"maas-api-6d9d8796ff-z7vvj\" (UID: \"3d7f6513-e1d8-4314-a352-478d97d59288\") " pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:11.932780 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:11.932707    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:12.007469 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.007432    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-openshift-service-ca\") pod \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") "
Sep 07 23:08:12.007469 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.007475    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-metrics-certs\") pod \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") "
Sep 07 23:08:12.007698 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.007536    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pps7t\" (UniqueName: \"kubernetes.io/projected/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-kube-api-access-pps7t\") pod \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") "
Sep 07 23:08:12.007698 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.007636    2582 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-maas-api-tls\") pod \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\" (UID: \"b32ea046-2283-48ee-a7a6-b2b42cbc92a5\") "
Sep 07 23:08:12.007890 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.007868    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "b32ea046-2283-48ee-a7a6-b2b42cbc92a5" (UID: "b32ea046-2283-48ee-a7a6-b2b42cbc92a5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 07 23:08:12.009758 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.009726    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "b32ea046-2283-48ee-a7a6-b2b42cbc92a5" (UID: "b32ea046-2283-48ee-a7a6-b2b42cbc92a5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:08:12.009851 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.009758    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-kube-api-access-pps7t" (OuterVolumeSpecName: "kube-api-access-pps7t") pod "b32ea046-2283-48ee-a7a6-b2b42cbc92a5" (UID: "b32ea046-2283-48ee-a7a6-b2b42cbc92a5"). InnerVolumeSpecName "kube-api-access-pps7t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 07 23:08:12.009851 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.009761    2582 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "b32ea046-2283-48ee-a7a6-b2b42cbc92a5" (UID: "b32ea046-2283-48ee-a7a6-b2b42cbc92a5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 07 23:08:12.057044 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.056996    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:12.109388 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.109254    2582 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pps7t\" (UniqueName: \"kubernetes.io/projected/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-kube-api-access-pps7t\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:12.109388 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.109307    2582 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-maas-api-tls\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:12.109388 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.109324    2582 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-openshift-service-ca\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:12.109388 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.109338    2582 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b32ea046-2283-48ee-a7a6-b2b42cbc92a5-metrics-certs\") on node \"ip-10-0-137-80.ec2.internal\" DevicePath \"\""
Sep 07 23:08:12.329255 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.329226    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"]
Sep 07 23:08:12.331053 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:08:12.331024    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d7f6513_e1d8_4314_a352_478d97d59288.slice/crio-672642febc8dc4c5a6cf655c226927d12b7b0cc8eea08e5d44eaf1a4209401f9 WatchSource:0}: Error finding container 672642febc8dc4c5a6cf655c226927d12b7b0cc8eea08e5d44eaf1a4209401f9: Status 404 returned error can't find the container with id 672642febc8dc4c5a6cf655c226927d12b7b0cc8eea08e5d44eaf1a4209401f9
Sep 07 23:08:12.380112 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.380074    2582 generic.go:356] "Generic (PLEG): container finished" podID="b32ea046-2283-48ee-a7a6-b2b42cbc92a5" containerID="402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1" exitCode=0
Sep 07 23:08:12.380307 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.380126    2582 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"
Sep 07 23:08:12.380307 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.380157    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b" event={"ID":"b32ea046-2283-48ee-a7a6-b2b42cbc92a5","Type":"ContainerDied","Data":"402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1"}
Sep 07 23:08:12.380307 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.380212    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b" event={"ID":"b32ea046-2283-48ee-a7a6-b2b42cbc92a5","Type":"ContainerDied","Data":"c134e0b7aeb5ffe74e3656df13deea9bfcb2614c297d950e36b90b43830b83fd"}
Sep 07 23:08:12.380307 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.380233    2582 scope.go:117] "RemoveContainer" containerID="402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1"
Sep 07 23:08:12.381557 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.381533    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj" event={"ID":"3d7f6513-e1d8-4314-a352-478d97d59288","Type":"ContainerStarted","Data":"672642febc8dc4c5a6cf655c226927d12b7b0cc8eea08e5d44eaf1a4209401f9"}
Sep 07 23:08:12.391086 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.391033    2582 scope.go:117] "RemoveContainer" containerID="402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1"
Sep 07 23:08:12.391544 ip-10-0-137-80 kubenswrapper[2582]: E0907 23:08:12.391511    2582 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1\": container with ID starting with 402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1 not found: ID does not exist" containerID="402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1"
Sep 07 23:08:12.391658 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.391549    2582 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1"} err="failed to get container status \"402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1\": rpc error: code = NotFound desc = could not find container \"402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1\": container with ID starting with 402fbf5e4b22311ebc8b3f09a0a3412fc6fabd26e35b0862834eebe2e7d59db1 not found: ID does not exist"
Sep 07 23:08:12.396150 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.396113    2582 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"]
Sep 07 23:08:12.398507 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:12.398483    2582 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-58d7778d67-dcq9b"]
Sep 07 23:08:13.389063 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:13.389028    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj" event={"ID":"3d7f6513-e1d8-4314-a352-478d97d59288","Type":"ContainerStarted","Data":"58b82f0cd5438e26b3474d6d9fd6a8815c04b54abce7f4633c1dc98a26dc1660"}
Sep 07 23:08:13.389619 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:13.389158    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:13.405582 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:13.405532    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj" podStartSLOduration=2.067313255 podStartE2EDuration="2.40551673s" totalImagesPullingTime="338.203475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-07 23:08:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:08:13.403548751 +0000 UTC m=+2159.869705746" watchObservedRunningTime="2026-09-07 23:08:13.40551673 +0000 UTC m=+2159.871673724"
Sep 07 23:08:14.090519 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:14.090483    2582 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b32ea046-2283-48ee-a7a6-b2b42cbc92a5" path="/var/lib/kubelet/pods/b32ea046-2283-48ee-a7a6-b2b42cbc92a5/volumes"
Sep 07 23:08:19.400620 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:19.400592    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d9d8796ff-z7vvj"
Sep 07 23:08:41.013533 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:41.013501    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7lcvs_8a7dacde-558e-48e5-b9e0-cea7b4d363c0/manager/0.log"
Sep 07 23:08:41.680974 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:41.680942    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-v2k2x_7b3bf17e-001d-4840-afff-e486c5503ba1/copy-manifests/0.log"
Sep 07 23:08:41.686266 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:41.686247    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-v2k2x_7b3bf17e-001d-4840-afff-e486c5503ba1/manager/0.log"
Sep 07 23:08:42.115008 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:42.114977    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-6d9d8796ff-z7vvj_3d7f6513-e1d8-4314-a352-478d97d59288/maas-api/0.log"
Sep 07 23:08:42.222539 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:42.222511    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29813685-lzn75_ef5395e7-5f04-4661-9296-fb0f13fff291/cleanup/0.log"
Sep 07 23:08:43.371407 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:43.371378    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-jbdcs_b4c23af5-92c5-43a0-b236-7c53ad497b9f/manager/0.log"
Sep 07 23:08:43.472937 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:43.472909    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-zn48v_76d16438-b115-4c83-803b-aa7fe8b53236/kuadrant-console-plugin/0.log"
Sep 07 23:08:43.582603 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:43.582561    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-w6glk_31c3211e-37c5-46ec-977e-18b69388eb7d/manager/0.log"
Sep 07 23:08:44.123193 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:44.123157    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn_f36ad099-80fa-4c53-a72a-9513fbf459e0/istio-proxy/0.log"
Sep 07 23:08:44.556695 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:44.556663    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-k86tk_2c570c8f-eb2a-407e-b198-45185ffbdb57/istio-proxy/1.log"
Sep 07 23:08:44.660975 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:44.660939    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r4nkj_e2d5d9f8-0bb1-440e-9376-7fab55da0edb/payload-pre-processing/0.log"
Sep 07 23:08:45.192043 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:45.192017    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-b2xkm_56233dc2-70d5-46f5-aeeb-390ac5bd6dd0/main/0.log"
Sep 07 23:08:45.299946 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:45.299914    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-4g5b8_f3f4c7c8-bef6-4dab-8748-c5b0cdce87de/main/0.log"
Sep 07 23:08:45.819880 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:45.819852    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-dk2qh_1d72cce6-e5a0-4ff1-9afc-8f934265deab/main/0.log"
Sep 07 23:08:53.509850 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:53.509807    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-22swc_a679bbf7-133d-425c-959d-8eb901ecdcf4/global-pull-secret-syncer/0.log"
Sep 07 23:08:53.651664 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:53.651627    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cj55w_2710c9fe-7828-47a8-82ea-2dc09f39d7d9/konnectivity-agent/0.log"
Sep 07 23:08:53.709358 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:53.709329    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-80.ec2.internal_bbd0e3824c495bf54791857c054ed650/haproxy/0.log"
Sep 07 23:08:57.868984 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:57.868953    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-5985564776-jbdcs_b4c23af5-92c5-43a0-b236-7c53ad497b9f/manager/0.log"
Sep 07 23:08:57.889256 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:57.889226    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-598f86468b-zn48v_76d16438-b115-4c83-803b-aa7fe8b53236/kuadrant-console-plugin/0.log"
Sep 07 23:08:57.926387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:08:57.926355    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5b65f94d4c-w6glk_31c3211e-37c5-46ec-977e-18b69388eb7d/manager/0.log"
Sep 07 23:09:00.266601 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:00.266568    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlgml_557864a2-d5cb-47bf-bfad-8538d655e028/node-exporter/0.log"
Sep 07 23:09:00.282495 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:00.282466    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlgml_557864a2-d5cb-47bf-bfad-8538d655e028/kube-rbac-proxy/0.log"
Sep 07 23:09:00.300870 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:00.300840    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-mlgml_557864a2-d5cb-47bf-bfad-8538d655e028/init-textfile/0.log"
Sep 07 23:09:03.758178 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:03.758149    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-64bbf7646b-sfpt2_19c511cb-0e33-4ab6-9071-9741b0feb41a/console/0.log"
Sep 07 23:09:04.172200 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.172163    2582 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"]
Sep 07 23:09:04.172514 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.172501    2582 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b32ea046-2283-48ee-a7a6-b2b42cbc92a5" containerName="maas-api"
Sep 07 23:09:04.172570 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.172516    2582 state_mem.go:107] "Deleted CPUSet assignment" podUID="b32ea046-2283-48ee-a7a6-b2b42cbc92a5" containerName="maas-api"
Sep 07 23:09:04.172611 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.172576    2582 memory_manager.go:368] "RemoveStaleState removing state" podUID="b32ea046-2283-48ee-a7a6-b2b42cbc92a5" containerName="maas-api"
Sep 07 23:09:04.175478 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.175461    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.177814 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.177793    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p8fk\"/\"kube-root-ca.crt\""
Sep 07 23:09:04.178848 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.178829    2582 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-8p8fk\"/\"default-dockercfg-ktzj4\""
Sep 07 23:09:04.178994 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.178832    2582 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-8p8fk\"/\"openshift-service-ca.crt\""
Sep 07 23:09:04.184655 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.184631    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"]
Sep 07 23:09:04.279139 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.279100    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-lib-modules\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.279139 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.279136    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-proc\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.279387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.279203    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-sys\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.279387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.279254    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwxtb\" (UniqueName: \"kubernetes.io/projected/f565b815-7149-43fd-a581-b96223ef0ed1-kube-api-access-cwxtb\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.279387 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.279355    2582 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-podres\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.379996 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.379955    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-sys\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380151 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380015    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwxtb\" (UniqueName: \"kubernetes.io/projected/f565b815-7149-43fd-a581-b96223ef0ed1-kube-api-access-cwxtb\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380151 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380068    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-podres\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380151 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380085    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-sys\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380151 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380133    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-lib-modules\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380151 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380151    2582 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-proc\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380365 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380187    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-podres\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380365 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380236    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-proc\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.380365 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.380252    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f565b815-7149-43fd-a581-b96223ef0ed1-lib-modules\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.387743 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.387719    2582 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwxtb\" (UniqueName: \"kubernetes.io/projected/f565b815-7149-43fd-a581-b96223ef0ed1-kube-api-access-cwxtb\") pod \"perf-node-gather-daemonset-skfxb\" (UID: \"f565b815-7149-43fd-a581-b96223ef0ed1\") " pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.485459 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.485363    2582 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:04.758471 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:04.758382    2582 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"]
Sep 07 23:09:04.762240 ip-10-0-137-80 kubenswrapper[2582]: W0907 23:09:04.762217    2582 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf565b815_7149_43fd_a581_b96223ef0ed1.slice/crio-b75c9129192f39ecd44fa7d0f7e806f8280adfed7059493db900dc1f3c3c3ecd WatchSource:0}: Error finding container b75c9129192f39ecd44fa7d0f7e806f8280adfed7059493db900dc1f3c3c3ecd: Status 404 returned error can't find the container with id b75c9129192f39ecd44fa7d0f7e806f8280adfed7059493db900dc1f3c3c3ecd
Sep 07 23:09:05.628862 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:05.628835    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tthjd_12799779-c22a-4ead-829f-e75c7eff4644/dns/0.log"
Sep 07 23:09:05.645758 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:05.645730    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-tthjd_12799779-c22a-4ead-829f-e75c7eff4644/kube-rbac-proxy/0.log"
Sep 07 23:09:05.660867 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:05.660838    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb" event={"ID":"f565b815-7149-43fd-a581-b96223ef0ed1","Type":"ContainerStarted","Data":"3062c1d7f29368908257308301f6ce019362010f9480a5490e1cd3d4eead1029"}
Sep 07 23:09:05.660998 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:05.660892    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:05.660998 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:05.660910    2582 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb" event={"ID":"f565b815-7149-43fd-a581-b96223ef0ed1","Type":"ContainerStarted","Data":"b75c9129192f39ecd44fa7d0f7e806f8280adfed7059493db900dc1f3c3c3ecd"}
Sep 07 23:09:05.664617 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:05.664597    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-62ddd_6075fe40-91f5-4f32-ada2-e3a3ed7c91fc/dns-node-resolver/0.log"
Sep 07 23:09:05.676090 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:05.676047    2582 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb" podStartSLOduration=1.6760319209999999 podStartE2EDuration="1.676031921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-07 23:09:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-07 23:09:05.675103341 +0000 UTC m=+2212.141260336" watchObservedRunningTime="2026-09-07 23:09:05.676031921 +0000 UTC m=+2212.142188915"
Sep 07 23:09:06.177611 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:06.177578    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-67fc699b4f-d9zgs_659fb193-89e8-49e2-9807-c5d133251ce8/registry/0.log"
Sep 07 23:09:06.219215 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:06.219189    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-fljbl_434fae71-2d34-4e71-b384-83aa51cf0a28/node-ca/0.log"
Sep 07 23:09:07.077327 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:07.077299    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn_f36ad099-80fa-4c53-a72a-9513fbf459e0/istio-proxy/0.log"
Sep 07 23:09:07.507009 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:07.506928    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-k86tk_2c570c8f-eb2a-407e-b198-45185ffbdb57/istio-proxy/0.log"
Sep 07 23:09:07.515755 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:07.515733    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-k86tk_2c570c8f-eb2a-407e-b198-45185ffbdb57/istio-proxy/1.log"
Sep 07 23:09:07.547794 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:07.547771    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r4nkj_e2d5d9f8-0bb1-440e-9376-7fab55da0edb/payload-pre-processing/0.log"
Sep 07 23:09:08.474489 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:08.474459    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-nr7g8_48ba9096-4eaf-4076-bba7-4e6cecd4d23c/serve-healthcheck-canary/0.log"
Sep 07 23:09:08.911527 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:08.911485    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-mkfjw_cd7b3adc-224b-4509-a86c-a892d692883d/insights-operator/0.log"
Sep 07 23:09:08.911920 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:08.911866    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-76bcd6ccf5-mkfjw_cd7b3adc-224b-4509-a86c-a892d692883d/insights-operator/1.log"
Sep 07 23:09:08.932572 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:08.932545    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxns_032352dd-0f32-454b-8363-69b634f412bd/kube-rbac-proxy/0.log"
Sep 07 23:09:08.950744 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:08.950723    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxns_032352dd-0f32-454b-8363-69b634f412bd/exporter/0.log"
Sep 07 23:09:08.968046 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:08.968023    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-8mxns_032352dd-0f32-454b-8363-69b634f412bd/extractor/0.log"
Sep 07 23:09:11.252631 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:11.252597    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-7lcvs_8a7dacde-558e-48e5-b9e0-cea7b4d363c0/manager/0.log"
Sep 07 23:09:11.618124 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:11.618097    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-v2k2x_7b3bf17e-001d-4840-afff-e486c5503ba1/manager/0.log"
Sep 07 23:09:11.634628 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:11.634598    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7d648d744d-v2k2x_7b3bf17e-001d-4840-afff-e486c5503ba1/copy-manifests/0.log"
Sep 07 23:09:11.675925 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:11.675894    2582 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-8p8fk/perf-node-gather-daemonset-skfxb"
Sep 07 23:09:12.717470 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:12.717442    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-575c877bb5-hrp27_dee80fb7-723a-406e-b9ed-fd3bc99a0335/manager/0.log"
Sep 07 23:09:12.759658 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:12.759628    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-hj5jj_9a332572-166b-4767-b357-7622d741c0c3/openshift-lws-operator/0.log"
Sep 07 23:09:14.116885 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:14.116853    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn_f36ad099-80fa-4c53-a72a-9513fbf459e0/istio-proxy/0.log"
Sep 07 23:09:14.586023 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:14.585991    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-k86tk_2c570c8f-eb2a-407e-b198-45185ffbdb57/istio-proxy/0.log"
Sep 07 23:09:14.595313 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:14.595265    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-k86tk_2c570c8f-eb2a-407e-b198-45185ffbdb57/istio-proxy/1.log"
Sep 07 23:09:14.619068 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:14.619042    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r4nkj_e2d5d9f8-0bb1-440e-9376-7fab55da0edb/payload-pre-processing/0.log"
Sep 07 23:09:16.355493 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:16.355406    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-dp2pn_f36ad099-80fa-4c53-a72a-9513fbf459e0/istio-proxy/0.log"
Sep 07 23:09:16.809220 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:16.809188    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-k86tk_2c570c8f-eb2a-407e-b198-45185ffbdb57/istio-proxy/0.log"
Sep 07 23:09:16.819852 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:16.819827    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-k86tk_2c570c8f-eb2a-407e-b198-45185ffbdb57/istio-proxy/1.log"
Sep 07 23:09:16.842860 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:16.842835    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-r4nkj_e2d5d9f8-0bb1-440e-9376-7fab55da0edb/payload-pre-processing/0.log"
Sep 07 23:09:18.006575 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:18.006544    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gvmg8_c92b7d60-6d95-472e-b96a-e29dea11ecf5/migrator/0.log"
Sep 07 23:09:18.024587 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:18.024560    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-fdbb8fdff-gvmg8_c92b7d60-6d95-472e-b96a-e29dea11ecf5/graceful-termination/0.log"
Sep 07 23:09:19.446690 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.446659    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx45q_cf3aa175-e534-4516-a436-d65670fc9198/kube-multus-additional-cni-plugins/0.log"
Sep 07 23:09:19.464882 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.464857    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx45q_cf3aa175-e534-4516-a436-d65670fc9198/egress-router-binary-copy/0.log"
Sep 07 23:09:19.481145 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.481123    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx45q_cf3aa175-e534-4516-a436-d65670fc9198/cni-plugins/0.log"
Sep 07 23:09:19.498156 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.498134    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx45q_cf3aa175-e534-4516-a436-d65670fc9198/bond-cni-plugin/0.log"
Sep 07 23:09:19.514896 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.514876    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx45q_cf3aa175-e534-4516-a436-d65670fc9198/routeoverride-cni/0.log"
Sep 07 23:09:19.531437 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.531419    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx45q_cf3aa175-e534-4516-a436-d65670fc9198/whereabouts-cni-bincopy/0.log"
Sep 07 23:09:19.547663 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.547645    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-kx45q_cf3aa175-e534-4516-a436-d65670fc9198/whereabouts-cni/0.log"
Sep 07 23:09:19.885442 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.885414    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-rjgxt_eef39637-cc2e-48ec-866a-101b11cd6329/kube-multus/0.log"
Sep 07 23:09:19.903628 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.903605    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4sqns_cc6804dc-8637-4596-8769-222cc116238e/network-metrics-daemon/0.log"
Sep 07 23:09:19.917063 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:19.917038    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-4sqns_cc6804dc-8637-4596-8769-222cc116238e/kube-rbac-proxy/0.log"
Sep 07 23:09:20.981532 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:20.981503    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/ovn-controller/0.log"
Sep 07 23:09:20.994220 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:20.994197    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/ovn-acl-logging/0.log"
Sep 07 23:09:21.012637 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:21.012563    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/kube-rbac-proxy-node/0.log"
Sep 07 23:09:21.028964 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:21.028898    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/kube-rbac-proxy-ovn-metrics/0.log"
Sep 07 23:09:21.043598 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:21.043575    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/northd/0.log"
Sep 07 23:09:21.059017 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:21.058999    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/nbdb/0.log"
Sep 07 23:09:21.074877 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:21.074856    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/sbdb/0.log"
Sep 07 23:09:21.128213 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:21.128191    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-h6c9s_eaaf6fa9-4605-446d-8948-77d89065d6e9/ovnkube-controller/0.log"
Sep 07 23:09:22.367237 ip-10-0-137-80 kubenswrapper[2582]: I0907 23:09:22.367207    2582 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-vrxqj_22f7133d-4d2a-4e79-acfc-44402fd0b947/network-check-target-container/0.log"